[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:05 dist-test-slave-1dwd krb5kdc[2646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:05 dist-test-slave-1dwd krb5kdc[2646](info): set up 2 sockets
Sep 26 07:27:05 dist-test-slave-1dwd krb5kdc[2646](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:27:06.477460 2639 mini_kdc.cc:121] Time spent starting KDC: real 3.225s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:06 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:06.645709 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43281
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:06.967583 2659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:06.967938 2659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:06.968112 2659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:06.975342 2659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:06.975476 2659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:06.975584 2659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:06.975688 2659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:06.975769 2659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:06.985785 2659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:06.987169 2659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:06.988507 2659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:07.001694 2664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:07.001663 2665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:07.002117 2667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:07.003957 2659 server_base.cc:1034] running on GCE node
I20260926 07:27:07.004880 2659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:07.006579 2659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:07.007885 2659 hybrid_clock.cc:648] HybridClock initialized: now 1790407627007789 us; error 85 us; skew 500 ppm
I20260926 07:27:07.008239 2659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:07 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:07.011180 2659 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:07.012558 2659 webserver.cc:460] Webserver started at http://127.2.147.254:45247/ using document root <none> and password file <none>
I20260926 07:27:07.013213 2659 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:07.013320 2659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:07.013561 2659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:07.016863 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "6f5102d820c749978bb78e3ad790647f"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "988e04a9cd8430ed2de0d6f3624a7b61"
server_key_iv: "aaf50ada6cc3e53429e0157975ac01ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.017558 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "6f5102d820c749978bb78e3ad790647f"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "988e04a9cd8430ed2de0d6f3624a7b61"
server_key_iv: "aaf50ada6cc3e53429e0157975ac01ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.022872 2659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:27:07.026139 2674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:07.027429 2659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:07.027595 2659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "6f5102d820c749978bb78e3ad790647f"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "988e04a9cd8430ed2de0d6f3624a7b61"
server_key_iv: "aaf50ada6cc3e53429e0157975ac01ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.027783 2659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:07.039120 2659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:07.042963 2659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:07.043283 2659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:07.062601 2659 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:43281
I20260926 07:27:07.062621 2725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:43281 every 8 connection(s)
I20260926 07:27:07.063998 2659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:07.067641 2726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:07.072322 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 2659
I20260926 07:27:07.072611 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:07.074360 2639 external_mini_cluster.cc:1428] Setting key b2a42e83e7ae1ac707cafcd94860514b
I20260926 07:27:07.077742 2726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f: Bootstrap starting.
I20260926 07:27:07.080919 2726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:07.081890 2726 log.cc:824] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:07.084482 2726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f: No bootstrap required, opened a new log
I20260926 07:27:07.094030 2726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } }
I20260926 07:27:07.094463 2726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:07.094585 2726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5102d820c749978bb78e3ad790647f, State: Initialized, Role: FOLLOWER
I20260926 07:27:07.095709 2726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } }
I20260926 07:27:07.096009 2726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:07.096169 2726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:07.096355 2726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:07.099040 2726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } }
I20260926 07:27:07.099632 2726 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f5102d820c749978bb78e3ad790647f; no voters:
I20260926 07:27:07.100257 2726 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:07.100402 2731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:07.102154 2726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:27:07 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:07.109864 2731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [term 1 LEADER]: Becoming Leader. State: Replica: 6f5102d820c749978bb78e3ad790647f, State: Running, Role: LEADER
I20260926 07:27:07.110510 2731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } }
I20260926 07:27:07.113766 2732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f5102d820c749978bb78e3ad790647f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } } }
I20260926 07:27:07.113804 2733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5102d820c749978bb78e3ad790647f. Latest consensus state: current_term: 1 leader_uuid: "6f5102d820c749978bb78e3ad790647f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5102d820c749978bb78e3ad790647f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43281 } } }
I20260926 07:27:07.114154 2732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:07.114195 2733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:07.121574 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:07.078112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.2.147.254:43281)
0926 07:27:07.078972 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:07.078987 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:07.080363 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 07:27:07.089653 (+ 9290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:07.089676 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:07.090152 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:07.090571 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:07.093271 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.095655 (+ 2384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.097488 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.098495 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.098669 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:07.106951 (+ 8282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:07.106965 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:07.106990 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:07.107091 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:07.113020 (+ 5929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:07.114188 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:07.114196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:07.114204 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:07.114448 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:07.115141 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:07.115145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:07.115148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:07.115938 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:07.119145 (+ 3207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:07.119566 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 07:27:07.119995 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":394,"threads_started":1}
I20260926 07:27:07.123967 2740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:07.129447 2740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:07.138602 2740 catalog_manager.cc:1347] Generated new cluster ID: 153fe167ae6b4575924aedb89af2224f
I20260926 07:27:07.138705 2740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:07.169806 2740 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:07.171720 2740 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:07.178431 2740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f5102d820c749978bb78e3ad790647f: Generated new TSK 0
I20260926 07:27:07.179461 2740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:07.222733 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:07.551882 2752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:07.552152 2752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:07.552245 2752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:07.559466 2752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:07.559577 2752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:07.559693 2752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:07.569386 2752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:07.571085 2752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:07.572417 2752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:07.585577 2758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:07.585660 2760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:07.585629 2757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:07.586436 2752 server_base.cc:1034] running on GCE node
I20260926 07:27:07.587020 2752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:07.588030 2752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:07.589285 2752 hybrid_clock.cc:648] HybridClock initialized: now 1790407627589226 us; error 53 us; skew 500 ppm
I20260926 07:27:07.589696 2752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:07 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:07.593355 2752 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:07.594823 2752 webserver.cc:460] Webserver started at http://127.2.147.193:41275/ using document root <none> and password file <none>
I20260926 07:27:07.595640 2752 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:07.595785 2752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:07.596091 2752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:07.598960 2752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "2939b06a872f4942806e59290008ec25"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "734cf9c361966ee31d0435ba0eb87f1a"
server_key_iv: "636ea465957101d8c65845f3ec3cd04e"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.599756 2752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "2939b06a872f4942806e59290008ec25"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "734cf9c361966ee31d0435ba0eb87f1a"
server_key_iv: "636ea465957101d8c65845f3ec3cd04e"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.604980 2752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:27:07.608274 2767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:07.609586 2752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:27:07.609717 2752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "2939b06a872f4942806e59290008ec25"
format_stamp: "Formatted at 2026-09-26 07:27:07 on dist-test-slave-1dwd"
server_key: "734cf9c361966ee31d0435ba0eb87f1a"
server_key_iv: "636ea465957101d8c65845f3ec3cd04e"
server_key_version: "encryptionkey@0"
I20260926 07:27:07.609880 2752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:07.634752 2752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:07.638604 2752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:07.638837 2752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:07.639744 2752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:07.640930 2752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:07.641005 2752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:07.641103 2752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:07.641145 2752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:07.675449 2752 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:41745
I20260926 07:27:07.675478 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:41745 every 8 connection(s)
I20260926 07:27:07.676797 2752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:07.679085 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 2752
I20260926 07:27:07.679358 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:07.680087 2639 external_mini_cluster.cc:1428] Setting key 5966d3e94bbc44c9372e1f9024925530
Sep 26 07:27:07 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:07.699406 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:07.679460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:51915 (local address 127.2.147.254:43281)
0926 07:27:07.679920 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:07.679926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:07.681837 (+ 1911us) server_negotiation.cc:407] Connection header received
0926 07:27:07.684260 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:07.684264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:07.684327 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:07.684410 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:07.686497 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.687321 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.689244 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.690027 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.690169 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:07.693753 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:07.693762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:07.693769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:07.693814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:07.696423 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:07.697309 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:07.697312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:07.697315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:07.697426 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:07.697806 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:07.697809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:07.697811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:07.698155 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:07.698281 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:07.698870 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 07:27:07.699024 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:27:07.700619 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:07.680211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43281 (local address 127.2.147.193:51915)
0926 07:27:07.681031 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:07.681073 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:07.683940 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:07.684700 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:07.684716 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:07.685524 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:07.686318 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:07.686344 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.687461 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.687468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:07.689091 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:07.689107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:07.690227 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:07.690232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:07.690414 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:07.691124 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:07.691146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:07.693539 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:07.696565 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:07.696577 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:07.696597 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:07.697204 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:07.697535 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:07.697538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:07.697540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:07.697714 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:07.698257 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:07.698263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:07.698556 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 07:27:07.698790 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 07:27:07.699311 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":293,"threads_started":1}
I20260926 07:27:07.702222 2880 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43281
I20260926 07:27:07.702555 2880 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:07.703307 2880 heartbeater.cc:511] Master 127.2.147.254:43281 requested a full tablet report, sending...
I20260926 07:27:07.705307 2691 ts_manager.cc:194] Registered new tserver with Master: 2939b06a872f4942806e59290008ec25 (127.2.147.193:41745)
I20260926 07:27:07.706738 2691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:51915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:07.720534 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:08.039163 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:08.039469 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:08.039559 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:08.046654 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:08.046746 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:08.046851 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:08.056031 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:08.057173 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:08.058368 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:08.071844 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:08.071885 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:08.072703 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:08.079891 2886 server_base.cc:1034] running on GCE node
I20260926 07:27:08.080497 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:08.081485 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:08.082701 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790407628082662 us; error 35 us; skew 500 ppm
I20260926 07:27:08.083004 2886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:08.086182 2886 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:08.087659 2886 webserver.cc:460] Webserver started at http://127.2.147.194:38519/ using document root <none> and password file <none>
I20260926 07:27:08.088372 2886 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:08.088522 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:08.088833 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:08.091681 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "8ccce59bb6c24c3bb43e8c6ecbae45c0"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "052e9e32b95e83e02e286e1ea56c2940"
server_key_iv: "8c78c864a663cdac593dd4b5e7a0b1bc"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.092448 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "8ccce59bb6c24c3bb43e8c6ecbae45c0"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "052e9e32b95e83e02e286e1ea56c2940"
server_key_iv: "8c78c864a663cdac593dd4b5e7a0b1bc"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.097344 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:27:08.100664 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.101830 2886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:27:08.101996 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "8ccce59bb6c24c3bb43e8c6ecbae45c0"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "052e9e32b95e83e02e286e1ea56c2940"
server_key_iv: "8c78c864a663cdac593dd4b5e7a0b1bc"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.102205 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:08.116077 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:08.120487 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:08.120779 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:08.121673 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:08.123020 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:08.123119 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.123265 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:08.123342 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.157662 2886 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:44107
I20260926 07:27:08.157717 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:44107 every 8 connection(s)
I20260926 07:27:08.158934 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:08.166020 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 2886
I20260926 07:27:08.166204 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:08.166623 2639 external_mini_cluster.cc:1428] Setting key 2f04b4189374a9ca040244348f46036a
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:08.179077 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.161934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:45811 (local address 127.2.147.254:43281)
0926 07:27:08.162105 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.162109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.163256 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 07:27:08.164969 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.164973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.165033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.165124 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.166596 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.167575 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.169231 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.169994 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.170145 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.173587 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.173592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.173595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.173624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.176158 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.176818 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.176825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.176828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.176956 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.177356 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.177362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.177365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.177614 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.177759 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.178477 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.178705 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:27:08.180017 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.162397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43281 (local address 127.2.147.194:45811)
0926 07:27:08.163076 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:08.163112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:08.164727 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:08.165327 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:08.165335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:08.165818 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:08.166447 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:08.166465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.167732 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.167736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:08.169091 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:08.169102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.170195 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.170200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:08.170405 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.171078 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:08.171098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:08.173365 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:08.176288 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:08.176299 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:08.176315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:08.176706 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:08.177060 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:08.177066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:08.177070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:08.177250 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:08.177736 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:08.177744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:08.178154 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 07:27:08.178411 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 07:27:08.178910 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260926 07:27:08.181406 3014 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43281
I20260926 07:27:08.181694 3014 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:27:08.182354 3014 heartbeater.cc:511] Master 127.2.147.254:43281 requested a full tablet report, sending...
I20260926 07:27:08.183790 2691 ts_manager.cc:194] Registered new tserver with Master: 8ccce59bb6c24c3bb43e8c6ecbae45c0 (127.2.147.194:44107)
I20260926 07:27:08.184588 2691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:45811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:08.205828 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:08.527074 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:08.527447 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:08.527626 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:08.534719 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:08.534852 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:08.535024 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:08.545017 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:08.546280 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:08.547672 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:08.560690 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:08.560768 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:08.560731 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:08.561462 3019 server_base.cc:1034] running on GCE node
I20260926 07:27:08.561980 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:08.562820 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:08.564007 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790407628563966 us; error 31 us; skew 500 ppm
I20260926 07:27:08.564323 3019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:08.571411 3019 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:08.572760 3019 webserver.cc:460] Webserver started at http://127.2.147.195:40375/ using document root <none> and password file <none>
I20260926 07:27:08.573453 3019 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:08.573596 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:08.573895 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:08.576813 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "2468cc925fd64c2b9005e6081b4cab6d"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "2f996b4f1786939f875ba3faea76b317"
server_key_iv: "51971eeaae2e1767bc940d6c3d70e15b"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.577579 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "2468cc925fd64c2b9005e6081b4cab6d"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "2f996b4f1786939f875ba3faea76b317"
server_key_iv: "51971eeaae2e1767bc940d6c3d70e15b"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.582670 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:27:08.586560 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.587952 3019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:27:08.588155 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "2468cc925fd64c2b9005e6081b4cab6d"
format_stamp: "Formatted at 2026-09-26 07:27:08 on dist-test-slave-1dwd"
server_key: "2f996b4f1786939f875ba3faea76b317"
server_key_iv: "51971eeaae2e1767bc940d6c3d70e15b"
server_key_version: "encryptionkey@0"
I20260926 07:27:08.588354 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:08.606400 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:08.610213 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:08.610497 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:08.611469 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:08.612766 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:08.612869 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.613024 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:08.613107 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:08.649099 3019 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:41183
I20260926 07:27:08.649130 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:41183 every 8 connection(s)
I20260926 07:27:08.650408 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:08.651299 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3019
I20260926 07:27:08.651510 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:08.651942 2639 external_mini_cluster.cc:1428] Setting key 05b341653dacb9b5ad7189d0c05c993d
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:08.670593 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.654188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:40547 (local address 127.2.147.254:43281)
0926 07:27:08.654428 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.654434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.654586 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:27:08.656662 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.656667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.656750 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.656857 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.658330 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.659061 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.660611 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.661020 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.661163 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.665073 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.665078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.665081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.665113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.667788 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.668691 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.668695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.668697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.668812 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.669201 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.669204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.669206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.669475 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.669581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.670185 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.670355 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 07:27:08.671725 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.653599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43281 (local address 127.2.147.195:40547)
0926 07:27:08.654441 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:08.654483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:08.656384 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:08.657107 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:08.657115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:08.657606 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:08.658174 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:08.658190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.659238 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.659246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:08.660464 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:08.660476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.661254 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.661263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:08.661509 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.662214 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:08.662242 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:08.664868 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:08.667922 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:08.667929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:08.667942 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:08.668569 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:08.668903 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:08.668906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:08.668909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:08.669105 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:08.669587 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:08.669592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:08.669888 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 07:27:08.670112 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:27:08.670488 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":333,"threads_started":1}
I20260926 07:27:08.673035 3147 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43281
I20260926 07:27:08.673296 3147 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:08.673938 3147 heartbeater.cc:511] Master 127.2.147.254:43281 requested a full tablet report, sending...
I20260926 07:27:08.675408 2691 ts_manager.cc:194] Registered new tserver with Master: 2468cc925fd64c2b9005e6081b4cab6d (127.2.147.195:41183)
I20260926 07:27:08.676522 2691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:40547
I20260926 07:27:08.680546 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 07:27:08.709595 2880 heartbeater.cc:503] Master 127.2.147.254:43281 was elected leader, sending a full tablet report...
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:08.723690 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.708430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.2.147.193:41745)
0926 07:27:08.709555 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.709565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.709603 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:27:08.709710 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.709719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.710031 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.710186 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.711721 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.712705 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.714032 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.714623 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.714754 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.718516 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.718526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.718540 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.718569 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.721205 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.721785 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.721791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.721794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.721910 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.722204 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.722207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.722209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.722588 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.722808 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.723044 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.723311 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":354,"threads_started":1}
W20260926 07:27:08.724951 2795 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38406
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:08.767342 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.754277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.2.147.193:41745)
0926 07:27:08.754490 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.754498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.754798 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:27:08.755001 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.755006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.755099 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.755206 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.756462 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.757123 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.758312 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.758720 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.758844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.762603 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.762608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.762611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.762640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.764979 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.765561 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.765567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.765571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.765689 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.766020 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.766026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.766030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.766266 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.766490 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.766781 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.766976 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:08.768738 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 2752
I20260926 07:27:08.780236 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.769647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.2.147.193:41745)
0926 07:27:08.769795 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.769798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.770098 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:27:08.770182 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.770185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.770235 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.770303 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.771332 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.771954 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.773167 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.773532 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.773617 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.776020 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.776024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.776027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.776051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.778292 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.778799 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.778802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.778804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.778918 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.779274 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.779278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.779280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.779484 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.779641 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.779865 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.779991 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:27:08.780877 2875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:08.861797 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 2752
I20260926 07:27:08.877889 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 2886
Sep 26 07:27:08 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:08.893144 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.878748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.2.147.194:44107)
0926 07:27:08.879370 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.879377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.879402 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:27:08.879461 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.879468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.879673 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.879800 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:08.880857 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.881819 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.882966 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:08.883650 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:08.883757 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:08.887728 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:08.887738 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:08.887753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:08.887784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:08.890451 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.891019 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.891025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.891028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.891146 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:08.891637 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:08.891640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:08.891642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:08.892015 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:08.892297 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:08.892554 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:27:08.892738 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":323,"threads_started":1}
I20260926 07:27:08.894471 3009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:08.975638 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 2886
I20260926 07:27:08.997764 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3019
Sep 26 07:27:09 dist-test-slave-1dwd krb5kdc[2646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:09.013335 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:08.998672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.2.147.195:41183)
0926 07:27:08.999317 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:08.999324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:08.999354 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:27:08.999412 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:08.999418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:08.999635 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:08.999758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:09.000861 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:09.001935 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:09.003355 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:09.003906 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:09.003997 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:09.007992 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:09.008002 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:09.008016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:09.008045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:09.010984 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:09.011681 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:09.011687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:09.011689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:09.011806 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:09.012104 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:09.012107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:09.012108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:09.012405 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:09.012652 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:09.012862 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 07:27:09.013060 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":301,"threads_started":1}
I20260926 07:27:09.014556 3142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:09.097611 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3019
I20260926 07:27:09.117982 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 2659
I20260926 07:27:09.119427 2721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:09.226243 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 2659
2026-09-26T07:27:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6025 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:09 dist-test-slave-1dwd krb5kdc[3168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:09 dist-test-slave-1dwd krb5kdc[3168](info): set up 2 sockets
Sep 26 07:27:09 dist-test-slave-1dwd krb5kdc[3168](info): commencing operation
krb5kdc: starting...
W20260926 07:27:10.292160 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:10 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:10.458431 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37927
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:37927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:10.764129 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:10.764483 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:10.764645 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:10.772056 3181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:10.772194 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:10.772277 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:10.772374 3181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:10.772439 3181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:10.781985 3181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:37927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:10.783294 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:10.784484 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:10.795746 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:10.795706 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:10.795781 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:10.796872 3181 server_base.cc:1034] running on GCE node
I20260926 07:27:10.797694 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:10.799014 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:10.800266 3181 hybrid_clock.cc:648] HybridClock initialized: now 1790407630800224 us; error 46 us; skew 500 ppm
I20260926 07:27:10.800617 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:10 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407630, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:10.803745 3181 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:10.805226 3181 webserver.cc:460] Webserver started at http://127.2.147.254:39479/ using document root <none> and password file <none>
I20260926 07:27:10.805894 3181 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:10.806013 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:10.806358 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:10.809183 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "b0f7680f6e704ca0868f823fc0a2fc21"
format_stamp: "Formatted at 2026-09-26 07:27:10 on dist-test-slave-1dwd"
server_key: "2eedab0a1b543779f8d4d85e7a4cd97a"
server_key_iv: "97a1194f6ab1f80202d29541a17833c9"
server_key_version: "encryptionkey@0"
I20260926 07:27:10.809864 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "b0f7680f6e704ca0868f823fc0a2fc21"
format_stamp: "Formatted at 2026-09-26 07:27:10 on dist-test-slave-1dwd"
server_key: "2eedab0a1b543779f8d4d85e7a4cd97a"
server_key_iv: "97a1194f6ab1f80202d29541a17833c9"
server_key_version: "encryptionkey@0"
I20260926 07:27:10.814909 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:27:10.818099 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:10.819388 3181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:27:10.819516 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "b0f7680f6e704ca0868f823fc0a2fc21"
format_stamp: "Formatted at 2026-09-26 07:27:10 on dist-test-slave-1dwd"
server_key: "2eedab0a1b543779f8d4d85e7a4cd97a"
server_key_iv: "97a1194f6ab1f80202d29541a17833c9"
server_key_version: "encryptionkey@0"
I20260926 07:27:10.819671 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:10.847288 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:10.851183 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:10.851433 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:10.869118 3181 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:37927
I20260926 07:27:10.869148 3247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:37927 every 8 connection(s)
I20260926 07:27:10.870411 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:10.874116 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3181
I20260926 07:27:10.874352 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:10.874505 3248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:10.875029 2639 external_mini_cluster.cc:1428] Setting key 04c78120317e1d53d2fef2745066f350
I20260926 07:27:10.886525 3248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21: Bootstrap starting.
Sep 26 07:27:10 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:10.889642 3248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:10.890836 3248 log.cc:824] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:10.893625 3248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21: No bootstrap required, opened a new log
I20260926 07:27:10.897255 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:10.876956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.2.147.254:37927)
0926 07:27:10.877731 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:10.877742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:10.877784 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:27:10.879381 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:10.879401 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:10.879786 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:10.880176 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:10.881590 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:10.882841 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:10.884109 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:10.884801 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:10.885007 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:10.888943 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:10.888951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:10.888964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:10.888998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:10.892152 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:10.892812 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:10.892819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:10.892829 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:10.892984 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:10.893368 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:10.893375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:10.893378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:10.894068 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:10.894411 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:10.895408 (+ 997us) server_negotiation.cc:299] Negotiation successful
0926 07:27:10.895885 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":333,"threads_started":1}
I20260926 07:27:10.902814 3248 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } }
I20260926 07:27:10.903134 3248 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:10.903259 3248 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f7680f6e704ca0868f823fc0a2fc21, State: Initialized, Role: FOLLOWER
I20260926 07:27:10.903803 3248 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } }
I20260926 07:27:10.904001 3248 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:10.904095 3248 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:10.904219 3248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:10.906424 3248 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } }
I20260926 07:27:10.906859 3248 leader_election.cc:302] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0f7680f6e704ca0868f823fc0a2fc21; no voters:
I20260926 07:27:10.907393 3248 leader_election.cc:288] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:10.907529 3253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:10.908221 3253 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [term 1 LEADER]: Becoming Leader. State: Replica: b0f7680f6e704ca0868f823fc0a2fc21, State: Running, Role: LEADER
I20260926 07:27:10.908727 3253 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } }
I20260926 07:27:10.909443 3248 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:10.912060 3254 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } } }
I20260926 07:27:10.912030 3255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f7680f6e704ca0868f823fc0a2fc21. Latest consensus state: current_term: 1 leader_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f7680f6e704ca0868f823fc0a2fc21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37927 } } }
I20260926 07:27:10.912555 3255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:10.913421 3254 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:10.913566 3262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:10.918702 3262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:10.927768 3262 catalog_manager.cc:1347] Generated new cluster ID: 730215792b9f454dbc2538e7c3bff0bc
I20260926 07:27:10.927882 3262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:10.953001 3262 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:10.954330 3262 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:10.969576 3262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0f7680f6e704ca0868f823fc0a2fc21: Generated new TSK 0
I20260926 07:27:10.970644 3262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:11.010957 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:11.333899 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:11.334194 3274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:11.334370 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:11.341550 3274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:11.341687 3274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:11.341883 3274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:11.351517 3274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:11.352849 3274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:11.354218 3274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:11.367178 3282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:11.367159 3280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:11.367578 3279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:11.368106 3274 server_base.cc:1034] running on GCE node
I20260926 07:27:11.368718 3274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:11.369602 3274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:11.370841 3274 hybrid_clock.cc:648] HybridClock initialized: now 1790407631370783 us; error 50 us; skew 500 ppm
I20260926 07:27:11.371167 3274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:11 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:11.374255 3274 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:11.375768 3274 webserver.cc:460] Webserver started at http://127.2.147.193:36675/ using document root <none> and password file <none>
I20260926 07:27:11.376430 3274 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:11.376572 3274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:11.376886 3274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:11.379788 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "8c42a621304e474daf1153fed88c4ad9"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "ed0420ed3685f26222dee28703115781"
server_key_iv: "7fb63ceed3fe64273ad67e539debe9e7"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.380563 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "8c42a621304e474daf1153fed88c4ad9"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "ed0420ed3685f26222dee28703115781"
server_key_iv: "7fb63ceed3fe64273ad67e539debe9e7"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.385480 3274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:27:11.388823 3289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.390147 3274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:11.390338 3274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "8c42a621304e474daf1153fed88c4ad9"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "ed0420ed3685f26222dee28703115781"
server_key_iv: "7fb63ceed3fe64273ad67e539debe9e7"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.390563 3274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:11.413045 3274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:11.417480 3274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:11.417764 3274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:11.418675 3274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:11.419919 3274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:11.419991 3274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.420163 3274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:11.420225 3274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.450922 3274 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45823
I20260926 07:27:11.450996 3401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45823 every 8 connection(s)
I20260926 07:27:11.452240 3274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:11.456333 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3274
I20260926 07:27:11.456550 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:11.457340 2639 external_mini_cluster.cc:1428] Setting key c72e0ac71cafd84808f4c8ad293b7dab
Sep 26 07:27:11 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:11.475282 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:11.454711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:45021 (local address 127.2.147.254:37927)
0926 07:27:11.455140 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:11.455145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:11.456062 (+ 917us) server_negotiation.cc:407] Connection header received
0926 07:27:11.458571 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:11.458575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:11.458636 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:11.458737 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:11.460778 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.461475 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.464026 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.464447 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.464539 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:11.468385 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:11.468394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:11.468402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:11.468444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:11.472098 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:11.472790 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:11.472804 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:11.472809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:11.473005 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:11.473420 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:11.473425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:11.473428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:11.473782 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:11.473936 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:11.474579 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 07:27:11.474819 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":240,"threads_started":1}
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:11.476070 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:11.455100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37927 (local address 127.2.147.193:45021)
0926 07:27:11.455869 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:11.455924 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:11.458274 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:11.458990 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:11.459001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:11.459727 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:11.460569 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:11.460598 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.461639 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.461648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:11.463882 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:11.463901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.464666 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.464674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:11.464992 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:11.465746 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:11.465770 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:11.468154 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:11.472230 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:11.472238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:11.472251 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:11.472658 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:11.473122 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:11.473125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:11.473127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:11.473299 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:11.473906 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:11.473912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:11.474211 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 07:27:11.474433 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 07:27:11.474813 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":260,"threads_started":1}
I20260926 07:27:11.477703 3402 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37927
I20260926 07:27:11.478014 3402 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:27:11.478868 3402 heartbeater.cc:511] Master 127.2.147.254:37927 requested a full tablet report, sending...
I20260926 07:27:11.481310 3213 ts_manager.cc:194] Registered new tserver with Master: 8c42a621304e474daf1153fed88c4ad9 (127.2.147.193:45823)
I20260926 07:27:11.483191 3213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:45021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:11.499588 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:11.812984 3408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:11.813298 3408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:11.813488 3408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:11.820418 3408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:11.820513 3408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:11.820623 3408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:11.829813 3408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:11.830979 3408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:11.832496 3408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:11.845026 3414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:11.845049 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:11.845363 3413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:11.845877 3408 server_base.cc:1034] running on GCE node
I20260926 07:27:11.846426 3408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:11.847409 3408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:11.848630 3408 hybrid_clock.cc:648] HybridClock initialized: now 1790407631848597 us; error 46 us; skew 500 ppm
I20260926 07:27:11.848928 3408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:11 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:11.851948 3408 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:11.853274 3408 webserver.cc:460] Webserver started at http://127.2.147.194:36009/ using document root <none> and password file <none>
I20260926 07:27:11.853880 3408 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:11.853972 3408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:11.854228 3408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:11.856932 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "148190d50f5c413b92a101edd6ce66eb"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "c0c2a0ee2a65b1fbbeb227a96341add9"
server_key_iv: "c6b8ab91849d188ae0f18fb8c01b5587"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.857627 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "148190d50f5c413b92a101edd6ce66eb"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "c0c2a0ee2a65b1fbbeb227a96341add9"
server_key_iv: "c6b8ab91849d188ae0f18fb8c01b5587"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.862437 3408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 07:27:11.865648 3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.866904 3408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:11.867020 3408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "148190d50f5c413b92a101edd6ce66eb"
format_stamp: "Formatted at 2026-09-26 07:27:11 on dist-test-slave-1dwd"
server_key: "c0c2a0ee2a65b1fbbeb227a96341add9"
server_key_iv: "c6b8ab91849d188ae0f18fb8c01b5587"
server_key_version: "encryptionkey@0"
I20260926 07:27:11.867180 3408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:11.883183 3408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:11.887135 3408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:11.887440 3408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:11.888361 3408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:11.889616 3408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:11.889690 3408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.889775 3408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:11.889818 3408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:11.922564 3408 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:40817
I20260926 07:27:11.922585 3535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:40817 every 8 connection(s)
I20260926 07:27:11.923918 3408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:11.925155 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3408
I20260926 07:27:11.925360 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:11.925892 2639 external_mini_cluster.cc:1428] Setting key eae88ac4004f9bd194980d83496b87f3
Sep 26 07:27:11 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
I20260926 07:27:11.945763 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:11.926680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:56215 (local address 127.2.147.254:37927)
0926 07:27:11.926862 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:11.926867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:11.928388 (+ 1521us) server_negotiation.cc:407] Connection header received
0926 07:27:11.931313 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:11.931317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:11.931377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:11.931461 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:11.933523 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.934241 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.935826 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.936216 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.936349 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:11.940222 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:11.940226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:11.940229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:11.940259 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:11.942753 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:11.943585 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:11.943589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:11.943591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:11.943711 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:11.944129 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:11.944133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:11.944135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:11.944369 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:11.944484 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:11.945094 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 07:27:11.945326 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:11.946794 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:11.927353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37927 (local address 127.2.147.194:56215)
0926 07:27:11.928196 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:11.928262 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:11.930963 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:11.931729 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:11.931743 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:11.932503 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:11.933357 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:11.933378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.934393 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.934398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:11.935685 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:11.935696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:11.936434 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:11.936442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:11.936672 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:11.937344 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:11.937366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:11.940006 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:11.942896 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:11.942907 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:11.942927 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:11.943475 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:11.943805 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:11.943808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:11.943810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:11.944018 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:11.944474 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:11.944479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:11.944774 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 07:27:11.944984 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 07:27:11.945387 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":340,"threads_started":1}
I20260926 07:27:11.948251 3536 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37927
I20260926 07:27:11.948592 3536 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:11.949497 3536 heartbeater.cc:511] Master 127.2.147.254:37927 requested a full tablet report, sending...
I20260926 07:27:11.951010 3213 ts_manager.cc:194] Registered new tserver with Master: 148190d50f5c413b92a101edd6ce66eb (127.2.147.194:40817)
I20260926 07:27:11.952337 3213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:56215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:11.966958 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:12.283582 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:12.283850 3541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:12.283943 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:12.290979 3541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:12.291090 3541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:12.291308 3541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:12.301144 3541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:12.302376 3541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:12.303638 3541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:12.316376 3546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:12.316430 3549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:12.316788 3547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:12.317188 3541 server_base.cc:1034] running on GCE node
I20260926 07:27:12.317673 3541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:12.318485 3541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:12.319702 3541 hybrid_clock.cc:648] HybridClock initialized: now 1790407632319660 us; error 54 us; skew 500 ppm
I20260926 07:27:12.320012 3541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:12.323067 3541 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:12.324519 3541 webserver.cc:460] Webserver started at http://127.2.147.195:33013/ using document root <none> and password file <none>
I20260926 07:27:12.325174 3541 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:12.325309 3541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:12.325598 3541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:12.328363 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "64b18b538be446d48908a72e1c1324ea"
format_stamp: "Formatted at 2026-09-26 07:27:12 on dist-test-slave-1dwd"
server_key: "3bab4a8569756d9959ac14aa45c9051d"
server_key_iv: "02628f4f5bc821bcaa98da694201d4ca"
server_key_version: "encryptionkey@0"
I20260926 07:27:12.329144 3541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "64b18b538be446d48908a72e1c1324ea"
format_stamp: "Formatted at 2026-09-26 07:27:12 on dist-test-slave-1dwd"
server_key: "3bab4a8569756d9959ac14aa45c9051d"
server_key_iv: "02628f4f5bc821bcaa98da694201d4ca"
server_key_version: "encryptionkey@0"
I20260926 07:27:12.333900 3541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:27:12.337095 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:12.338284 3541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:27:12.338446 3541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "64b18b538be446d48908a72e1c1324ea"
format_stamp: "Formatted at 2026-09-26 07:27:12 on dist-test-slave-1dwd"
server_key: "3bab4a8569756d9959ac14aa45c9051d"
server_key_iv: "02628f4f5bc821bcaa98da694201d4ca"
server_key_version: "encryptionkey@0"
I20260926 07:27:12.338629 3541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:12.351722 3541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:12.355521 3541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:12.355813 3541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:12.356727 3541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:12.357976 3541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:12.358079 3541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:12.358232 3541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:12.358305 3541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:12.390189 3541 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:46295
I20260926 07:27:12.390228 3668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:46295 every 8 connection(s)
I20260926 07:27:12.391615 3541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:12.392678 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3541
I20260926 07:27:12.392858 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:12.393462 2639 external_mini_cluster.cc:1428] Setting key 118160af435f47b373863e806fe32f37
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:12.411602 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.394313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:40127 (local address 127.2.147.254:37927)
0926 07:27:12.394501 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.394506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.395992 (+ 1486us) server_negotiation.cc:407] Connection header received
0926 07:27:12.398332 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.398335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.398399 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.398476 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.399989 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.400694 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.402177 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.402607 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.402707 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.405864 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.405868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.405871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.405898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.408693 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.409392 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.409395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.409397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.409501 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.409946 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.409949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.409951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.410248 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.410375 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.411059 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.411295 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:12.412513 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.394893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37927 (local address 127.2.147.195:40127)
0926 07:27:12.395811 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.395871 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.398087 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.398681 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.398688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.399202 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:12.399851 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.399869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.400810 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.400813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.402026 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.402037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.402796 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.402800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.402992 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.403651 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:12.403671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:12.405686 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:12.408821 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:12.408829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:12.408848 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:12.409285 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:12.409601 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:12.409604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:12.409607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:12.409849 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:12.410382 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:12.410387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:12.410730 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.410951 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.411439 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":393,"threads_started":1}
I20260926 07:27:12.413843 3669 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37927
I20260926 07:27:12.414167 3669 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:12.414878 3669 heartbeater.cc:511] Master 127.2.147.254:37927 requested a full tablet report, sending...
I20260926 07:27:12.416270 3213 ts_manager.cc:194] Registered new tserver with Master: 64b18b538be446d48908a72e1c1324ea (127.2.147.195:46295)
I20260926 07:27:12.417012 3213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:40127
I20260926 07:27:12.420715 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:12.438961 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.427098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.2.147.254:37927)
0926 07:27:12.427328 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.427337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.427775 (+ 438us) server_negotiation.cc:407] Connection header received
0926 07:27:12.427955 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.427960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.428029 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.428134 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.429542 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.430276 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.431618 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.432037 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.432166 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.434354 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.434360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.434363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.434390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.436577 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.437163 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.437170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.437173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.437298 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.437642 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.437648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.437653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.437931 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.438185 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.438468 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.438664 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:12.445194 3213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:12.452724 3213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:12.486338 3402 heartbeater.cc:503] Master 127.2.147.254:37927 was elected leader, sending a full tablet report...
I20260926 07:27:12.488143 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.464365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.2.147.195:46295)
0926 07:27:12.477259 (+ 12894us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.477272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.478950 (+ 1678us) server_negotiation.cc:407] Connection header received
0926 07:27:12.479344 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.479363 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.479617 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.479829 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:12.480721 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.482173 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.484409 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.485910 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.486031 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.486301 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.486912 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.487084 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12645,"thread_start_us":12913,"threads_started":1}
I20260926 07:27:12.488346 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.464514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.2.147.194:40817)
0926 07:27:12.477885 (+ 13371us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.477891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.479244 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 07:27:12.480088 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.480099 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.480380 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.480536 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:12.481916 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.483013 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.485093 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.486627 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.486762 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.486942 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.487913 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.488077 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13226,"thread_start_us":13132,"threads_started":1}
I20260926 07:27:12.488708 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.464007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:46295 (local address 127.0.0.1:54536)
0926 07:27:12.478760 (+ 14753us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.478792 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.479050 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.480131 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.480139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.480163 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:12.480523 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.480535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.482294 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.482297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.484265 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.484278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.486429 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.486436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.486637 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.486690 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.488144 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.488342 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14534,"thread_start_us":15173,"threads_started":1}
I20260926 07:27:12.489135 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.479506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45823 (local address 127.0.0.1:46016)
0926 07:27:12.480037 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.480055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.480147 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.481173 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.481177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.481202 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:12.481736 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.481748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.484335 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.484343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.486351 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.486363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.488229 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.488250 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.488407 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.488440 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.488507 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.488641 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":2,"thread_start_us":344,"threads_started":1}
I20260926 07:27:12.489255 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.464326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:40817 (local address 127.0.0.1:50250)
0926 07:27:12.479117 (+ 14791us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.479134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.479750 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.481138 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.481142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.481167 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:12.481742 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.481750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.483145 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.483151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.484942 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.484952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.487572 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.487579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.487725 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.487758 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.488743 (+ 985us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.488891 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14679,"thread_start_us":14104,"threads_started":1}
I20260926 07:27:12.489554 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.479874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.2.147.193:45823)
0926 07:27:12.480622 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.480631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.480657 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:27:12.480757 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.480763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.480976 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.481462 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:12.483152 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.484195 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.486545 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.487993 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.488782 (+ 789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.489009 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.489081 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.489234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":1258,"threads_started":1}
I20260926 07:27:12.492482 3604 tablet_service.cc:1467] Processing CreateTablet for tablet 03670f2f89e644c4b109cd283b6b515d (DEFAULT_TABLE table=test-table [id=25351e8e0434443493f886d81f1145c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:12.492652 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 03670f2f89e644c4b109cd283b6b515d (DEFAULT_TABLE table=test-table [id=25351e8e0434443493f886d81f1145c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:12.492651 3337 tablet_service.cc:1467] Processing CreateTablet for tablet 03670f2f89e644c4b109cd283b6b515d (DEFAULT_TABLE table=test-table [id=25351e8e0434443493f886d81f1145c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:12.493821 3604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03670f2f89e644c4b109cd283b6b515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:12.493949 3471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03670f2f89e644c4b109cd283b6b515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:12.493954 3337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03670f2f89e644c4b109cd283b6b515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:12.505396 3687 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Bootstrap starting.
I20260926 07:27:12.508316 3688 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Bootstrap starting.
I20260926 07:27:12.509312 3687 tablet_bootstrap.cc:654] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:12.509405 3689 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Bootstrap starting.
I20260926 07:27:12.510720 3687 log.cc:824] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:12.512629 3688 tablet_bootstrap.cc:654] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:12.513118 3689 tablet_bootstrap.cc:654] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:12.513892 3688 log.cc:824] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:12.514359 3689 log.cc:824] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:12.514678 3687 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: No bootstrap required, opened a new log
I20260926 07:27:12.515021 3687 ts_tablet_manager.cc:1397] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 07:27:12.517021 3688 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: No bootstrap required, opened a new log
I20260926 07:27:12.517282 3688 ts_tablet_manager.cc:1397] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:27:12.517306 3689 tablet_bootstrap.cc:492] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: No bootstrap required, opened a new log
I20260926 07:27:12.517534 3689 ts_tablet_manager.cc:1397] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 07:27:12.524653 3688 raft_consensus.cc:348] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.525044 3688 raft_consensus.cc:374] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:12.525151 3688 raft_consensus.cc:729] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c42a621304e474daf1153fed88c4ad9, State: Initialized, Role: FOLLOWER
I20260926 07:27:12.525749 3688 consensus_queue.cc:260] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.525885 3687 raft_consensus.cc:348] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.526362 3687 raft_consensus.cc:374] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:12.526487 3687 raft_consensus.cc:729] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b18b538be446d48908a72e1c1324ea, State: Initialized, Role: FOLLOWER
I20260926 07:27:12.527249 3687 consensus_queue.cc:260] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.527731 3688 ts_tablet_manager.cc:1428] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260926 07:27:12.527872 3689 raft_consensus.cc:348] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.528450 3689 raft_consensus.cc:374] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:12.528602 3689 raft_consensus.cc:729] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148190d50f5c413b92a101edd6ce66eb, State: Initialized, Role: FOLLOWER
I20260926 07:27:12.529326 3689 consensus_queue.cc:260] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.530577 3536 heartbeater.cc:503] Master 127.2.147.254:37927 was elected leader, sending a full tablet report...
I20260926 07:27:12.531299 3689 ts_tablet_manager.cc:1428] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260926 07:27:12.532176 3669 heartbeater.cc:503] Master 127.2.147.254:37927 was elected leader, sending a full tablet report...
I20260926 07:27:12.532804 3687 ts_tablet_manager.cc:1428] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260926 07:27:12.643152 3670 tablet.cc:2369] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:12.675871 3537 tablet.cc:2369] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:12.704550 3403 tablet.cc:2369] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:12.772085 3694 raft_consensus.cc:482] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:12.772408 3694 raft_consensus.cc:504] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.774519 3694 leader_election.cc:288] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148190d50f5c413b92a101edd6ce66eb (127.2.147.194:40817), 8c42a621304e474daf1153fed88c4ad9 (127.2.147.193:45823)
I20260926 07:27:12.779798 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.774817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:60177 (local address 127.2.147.193:45823)
0926 07:27:12.775175 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.775182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.775223 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:27:12.775297 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.775300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.775354 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.775457 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:12.775825 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.776500 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.778091 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.778848 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.779034 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.779102 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.779415 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.779537 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260926 07:27:12.779811 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.774381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45823 (local address 127.2.147.195:60177)
0926 07:27:12.774698 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.774718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.774840 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.775498 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.775505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.775530 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:12.775689 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.775697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.776624 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.776632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.777926 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.777939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.779033 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.779038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.779256 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.779289 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.779388 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.779514 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 07:27:12.780391 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.774975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:42491 (local address 127.2.147.194:40817)
0926 07:27:12.775175 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.775182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.775600 (+ 418us) server_negotiation.cc:407] Connection header received
0926 07:27:12.775798 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.775801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.775855 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.775948 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:12.776405 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.777068 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.778675 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.779444 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.779622 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.779688 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.780023 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.780144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:27:12.781095 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.774750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:40817 (local address 127.2.147.195:42491)
0926 07:27:12.775495 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:12.775508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:12.775662 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:12.775959 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:12.775963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:12.775982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:12.776259 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.776266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.777218 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.777223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.778522 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:12.778532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.779641 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.779646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:12.779819 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.779841 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:27:12.780572 (+ 731us) client_negotiation.cc:240] Negotiation successful
0926 07:27:12.780720 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":1,"thread_start_us":431,"threads_started":1}
I20260926 07:27:12.780879 3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03670f2f89e644c4b109cd283b6b515d" candidate_uuid: "64b18b538be446d48908a72e1c1324ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c42a621304e474daf1153fed88c4ad9" is_pre_election: true
I20260926 07:27:12.781454 3357 raft_consensus.cc:2454] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b18b538be446d48908a72e1c1324ea in term 0.
I20260926 07:27:12.782073 3557 leader_election.cc:302] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b18b538be446d48908a72e1c1324ea, 8c42a621304e474daf1153fed88c4ad9; no voters:
I20260926 07:27:12.782083 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03670f2f89e644c4b109cd283b6b515d" candidate_uuid: "64b18b538be446d48908a72e1c1324ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148190d50f5c413b92a101edd6ce66eb" is_pre_election: true
I20260926 07:27:12.782474 3694 raft_consensus.cc:2781] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:12.782595 3694 raft_consensus.cc:482] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:12.782714 3694 raft_consensus.cc:3037] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:12.782526 3491 raft_consensus.cc:2454] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b18b538be446d48908a72e1c1324ea in term 0.
I20260926 07:27:12.785073 3694 raft_consensus.cc:504] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.785668 3694 leader_election.cc:288] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [CANDIDATE]: Term 1 election: Requested vote from peers 148190d50f5c413b92a101edd6ce66eb (127.2.147.194:40817), 8c42a621304e474daf1153fed88c4ad9 (127.2.147.193:45823)
I20260926 07:27:12.786024 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03670f2f89e644c4b109cd283b6b515d" candidate_uuid: "64b18b538be446d48908a72e1c1324ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148190d50f5c413b92a101edd6ce66eb"
I20260926 07:27:12.786164 3357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03670f2f89e644c4b109cd283b6b515d" candidate_uuid: "64b18b538be446d48908a72e1c1324ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c42a621304e474daf1153fed88c4ad9"
I20260926 07:27:12.786240 3491 raft_consensus.cc:3037] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:12.786350 3357 raft_consensus.cc:3037] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:12.789362 3357 raft_consensus.cc:2454] T 03670f2f89e644c4b109cd283b6b515d P 8c42a621304e474daf1153fed88c4ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b18b538be446d48908a72e1c1324ea in term 1.
I20260926 07:27:12.789359 3491 raft_consensus.cc:2454] T 03670f2f89e644c4b109cd283b6b515d P 148190d50f5c413b92a101edd6ce66eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b18b538be446d48908a72e1c1324ea in term 1.
I20260926 07:27:12.789736 3557 leader_election.cc:302] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b18b538be446d48908a72e1c1324ea, 8c42a621304e474daf1153fed88c4ad9; no voters:
I20260926 07:27:12.790022 3694 raft_consensus.cc:2781] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:12.790444 3694 raft_consensus.cc:686] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [term 1 LEADER]: Becoming Leader. State: Replica: 64b18b538be446d48908a72e1c1324ea, State: Running, Role: LEADER
I20260926 07:27:12.790930 3694 consensus_queue.cc:237] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } }
I20260926 07:27:12.795979 3213 catalog_manager.cc:5495] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b18b538be446d48908a72e1c1324ea (127.2.147.195). New cstate: current_term: 1 leader_uuid: "64b18b538be446d48908a72e1c1324ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b18b538be446d48908a72e1c1324ea" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148190d50f5c413b92a101edd6ce66eb" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c42a621304e474daf1153fed88c4ad9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45823 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:12.845942 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.832005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.2.147.193:45823)
0926 07:27:12.832226 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.832230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.832319 (+ 89us) server_negotiation.cc:407] Connection header received
0926 07:27:12.832458 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.832462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.832524 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.832608 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.833994 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.834789 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.836572 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.836959 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.837102 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.840785 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.840799 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.840815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.840849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.843723 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.844313 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.844319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.844322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.844440 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.844746 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.844749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.844751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.845099 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.845315 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.845519 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.845669 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:12.891975 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.877959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.2.147.193:45823)
0926 07:27:12.878151 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.878158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.878543 (+ 385us) server_negotiation.cc:407] Connection header received
0926 07:27:12.878690 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.878695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.878771 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.878896 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.880219 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.881319 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.882591 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.883231 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.883361 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.886910 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.886915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.886918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.886948 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.889568 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.890083 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.890087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.890089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.890198 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.890578 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.890593 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.890595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.890837 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.891497 (+ 660us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.891593 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.891743 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:27:12 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:12.936808 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.922856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.2.147.193:45823)
0926 07:27:12.923055 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.923064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.923495 (+ 431us) server_negotiation.cc:407] Connection header received
0926 07:27:12.923731 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.923737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.923811 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.923918 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.925215 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.925891 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.927111 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.927596 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.927729 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.931485 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.931490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.931493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.931522 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.933905 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.934941 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.934948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.934955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.935080 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.935474 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.935481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.935484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.935755 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.935996 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.936292 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.936507 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 07:27:12.938315 3317 tablet_service.cc:2908] Rejecting scan request for tablet 03670f2f89e644c4b109cd283b6b515d: Uninitialized: clean time has not yet been initialized
I20260926 07:27:12.941654 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3274
I20260926 07:27:12.953209 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:12.942457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.2.147.193:45823)
0926 07:27:12.942593 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:12.942597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:12.942926 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:27:12.943050 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:12.943053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:12.943115 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:12.943191 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:12.944336 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.944967 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.946063 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:12.946456 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:12.946554 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:12.948565 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:12.948569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:12.948572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:12.948597 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:12.951082 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.951664 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.951667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.951669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.951776 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:12.952066 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:12.952069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:12.952071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:12.952329 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:12.952525 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:12.952818 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:27:12.952971 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:27:12.953898 3397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:13.042176 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3274
I20260926 07:27:13.058725 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3408
Sep 26 07:27:13 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:13.072986 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:13.059522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.2.147.194:40817)
0926 07:27:13.059690 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:13.059697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:13.059793 (+ 96us) server_negotiation.cc:407] Connection header received
0926 07:27:13.059999 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:13.060009 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:13.060069 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:13.060179 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:13.061298 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:13.062000 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:13.063370 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:13.063763 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:13.063861 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:13.067525 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:13.067538 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:13.067554 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:13.067584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:13.070108 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:13.070720 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:13.070726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:13.070729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:13.070909 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:13.071263 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:13.071269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:13.071272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:13.071774 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:13.072099 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:13.072578 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 07:27:13.072740 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:27:13.073621 3531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:13.164637 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3408
I20260926 07:27:13.180006 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:13.179579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45823 (local address 127.2.147.195:56823)
0926 07:27:13.179749 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:13.179860 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:45823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:27:13.180390 3557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.193:45823: connect: Connection refused (error 111)
I20260926 07:27:13.181856 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:13.181442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45823 (local address 127.2.147.195:53847)
0926 07:27:13.181605 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:13.181701 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:45823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:27:13.182017 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3541
W20260926 07:27:13.182065 3557 consensus_peers.cc:487] T 03670f2f89e644c4b109cd283b6b515d P 64b18b538be446d48908a72e1c1324ea -> Peer 8c42a621304e474daf1153fed88c4ad9 (127.2.147.193:45823): Couldn't send request to peer 8c42a621304e474daf1153fed88c4ad9. Status: Network error: Client connection negotiation failed: client connection to 127.2.147.193:45823: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 26 07:27:13 dist-test-slave-1dwd krb5kdc[3168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:13.195598 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:13.182671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.2.147.195:46295)
0926 07:27:13.183050 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:13.183054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:13.183073 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:27:13.183124 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:13.183128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:13.183186 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:13.183334 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:13.184476 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:13.185138 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:13.186347 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:13.186722 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:13.186809 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:13.190526 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:13.190537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:13.190552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:13.190579 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:13.193294 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:13.193854 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:13.193860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:13.193862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:13.193981 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:13.194294 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:13.194297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:13.194299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:13.194597 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:13.194873 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:13.195020 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:27:13.195283 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":193,"threads_started":1}
I20260926 07:27:13.196220 3664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:13.285481 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3541
I20260926 07:27:13.301877 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3181
I20260926 07:27:13.303010 3243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:13.408381 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3181
2026-09-26T07:27:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4191 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:13 dist-test-slave-1dwd krb5kdc[3718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:13 dist-test-slave-1dwd krb5kdc[3718](info): set up 2 sockets
Sep 26 07:27:13 dist-test-slave-1dwd krb5kdc[3718](info): commencing operation
krb5kdc: starting...
W20260926 07:27:14.487998 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:14 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:14.662101 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34681
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:14.970381 3731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:14.970646 3731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:14.970762 3731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:14.977775 3731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:14.977866 3731 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:27:14.977968 3731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:14.978101 3731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:14.978178 3731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:14.978291 3731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:14.987839 3731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:14.989094 3731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:14.990409 3731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:15.002022 3736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:15.001976 3737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:15.001994 3739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:15.002699 3731 server_base.cc:1034] running on GCE node
I20260926 07:27:15.003516 3731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:15.004822 3731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:15.006062 3731 hybrid_clock.cc:648] HybridClock initialized: now 1790407635006011 us; error 31 us; skew 500 ppm
I20260926 07:27:15.006402 3731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:15 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:15.012989 3731 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:15.014406 3731 webserver.cc:460] Webserver started at http://127.2.147.254:39847/ using document root <none> and password file <none>
I20260926 07:27:15.015025 3731 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:15.015115 3731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:15.015436 3731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:15.018246 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "0e3ad818fd2049c5bfeb77cbf9450698"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "29846bc5ad0b00468988dd73366d9ca5"
server_key_iv: "fd41238bcbf17ac5afa339071744799a"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.018952 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "0e3ad818fd2049c5bfeb77cbf9450698"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "29846bc5ad0b00468988dd73366d9ca5"
server_key_iv: "fd41238bcbf17ac5afa339071744799a"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.023694 3731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:27:15.026808 3746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:15.028136 3731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:15.028307 3731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "0e3ad818fd2049c5bfeb77cbf9450698"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "29846bc5ad0b00468988dd73366d9ca5"
server_key_iv: "fd41238bcbf17ac5afa339071744799a"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.028512 3731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:15.045192 3731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:15.049351 3731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:15.049588 3731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:15.070699 3731 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34681
I20260926 07:27:15.070720 3807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34681 every 8 connection(s)
I20260926 07:27:15.072090 3731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:15.075795 3808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:15.077996 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3731
I20260926 07:27:15.078210 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:15.078972 2639 external_mini_cluster.cc:1428] Setting key 03ae41ef87212a6ca3a2f7591c47b68f
I20260926 07:27:15.088464 3808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Bootstrap starting.
I20260926 07:27:15.092406 3808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:15.093745 3808 log.cc:824] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:27:15 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:15.097311 3808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: No bootstrap required, opened a new log
I20260926 07:27:15.103856 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:15.080939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.2.147.254:34681)
0926 07:27:15.081727 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:15.081739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:15.081779 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:27:15.083943 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:15.083966 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:15.084340 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:15.084671 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:15.086390 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.087786 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.089473 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.090166 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.090295 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:15.095443 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:15.095454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:15.095474 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:15.095518 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:15.099340 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.099995 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.100003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.100013 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.100246 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.100585 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.100590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.100593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.101300 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:15.101604 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:15.101913 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:27:15.102358 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":373,"threads_started":1}
I20260926 07:27:15.108357 3808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } }
I20260926 07:27:15.108703 3808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:15.108807 3808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3ad818fd2049c5bfeb77cbf9450698, State: Initialized, Role: FOLLOWER
I20260926 07:27:15.109380 3808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } }
I20260926 07:27:15.109614 3808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:15.109771 3808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:15.109958 3808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:15.113267 3808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } }
I20260926 07:27:15.113858 3808 leader_election.cc:302] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e3ad818fd2049c5bfeb77cbf9450698; no voters:
I20260926 07:27:15.114485 3808 leader_election.cc:288] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:15.114708 3813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:15.118166 3813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [term 1 LEADER]: Becoming Leader. State: Replica: 0e3ad818fd2049c5bfeb77cbf9450698, State: Running, Role: LEADER
I20260926 07:27:15.118237 3808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:15.118891 3813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } }
I20260926 07:27:15.123183 3815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3ad818fd2049c5bfeb77cbf9450698. Latest consensus state: current_term: 1 leader_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } } }
I20260926 07:27:15.123612 3815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:15.125869 3814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3ad818fd2049c5bfeb77cbf9450698" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34681 } } }
I20260926 07:27:15.126214 3814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:15.127174 3819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:15.132658 3819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:15.140317 3819 catalog_manager.cc:1347] Generated new cluster ID: 6ae37cb276904147b7fcf0ccf0dc1108
I20260926 07:27:15.140419 3819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:15.157250 3819 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:15.159173 3819 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:15.169565 3819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Generated new TSK 0
I20260926 07:27:15.170673 3819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:15.212735 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34681
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:15.530524 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:15.530802 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:15.530898 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:15.538100 3834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:15.538196 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:15.538286 3834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:15.538364 3834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:15.547734 3834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:15.548892 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:15.550115 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:15.563135 3839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:15.563242 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:15.563162 3840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:15.564172 3834 server_base.cc:1034] running on GCE node
I20260926 07:27:15.564749 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:15.565666 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:15.566891 3834 hybrid_clock.cc:648] HybridClock initialized: now 1790407635566835 us; error 43 us; skew 500 ppm
I20260926 07:27:15.567138 3834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:15 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:15.570195 3834 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:15.571657 3834 webserver.cc:460] Webserver started at http://127.2.147.193:44887/ using document root <none> and password file <none>
I20260926 07:27:15.572335 3834 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:15.572458 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:15.572759 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:15.575582 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "5ec16586d401422fa8fdfa7acc81f2de"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "5aa877b3e2994c9eeb400827c957d524"
server_key_iv: "beb3e7f28ed431a9e3e63960d2123d7b"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.576363 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "5ec16586d401422fa8fdfa7acc81f2de"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "5aa877b3e2994c9eeb400827c957d524"
server_key_iv: "beb3e7f28ed431a9e3e63960d2123d7b"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.581430 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:27:15.584749 3849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:15.586024 3834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:15.586202 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "5ec16586d401422fa8fdfa7acc81f2de"
format_stamp: "Formatted at 2026-09-26 07:27:15 on dist-test-slave-1dwd"
server_key: "5aa877b3e2994c9eeb400827c957d524"
server_key_iv: "beb3e7f28ed431a9e3e63960d2123d7b"
server_key_version: "encryptionkey@0"
I20260926 07:27:15.586407 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:15.622092 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:15.626015 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:15.626298 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:15.629324 3834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:15.629434 3834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:15.629578 3834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:15.629652 3834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:27:15 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:15.655068 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:15.630707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.193:43943)
0926 07:27:15.631598 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:15.631656 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:15.633724 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:15.634514 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:15.634524 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:15.635048 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:15.635889 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:15.635918 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.637461 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.637468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:15.639842 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:15.639860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.641006 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.641011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:15.641208 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:15.642104 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:15.642144 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:15.645250 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:15.649712 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:15.649725 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:15.649745 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:15.650240 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:15.650742 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:15.650752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:15.650755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:15.651127 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:15.651920 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:15.651931 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:15.652439 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 07:27:15.652753 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 07:27:15.653340 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":405,"threads_started":1}
I20260926 07:27:15.655761 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:15.629959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43943 (local address 127.2.147.254:34681)
0926 07:27:15.630519 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:15.630525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:15.631772 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 07:27:15.633988 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:15.633998 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:15.634103 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:15.634233 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:15.636150 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.637228 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.640004 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.640794 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.640920 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:15.645521 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:15.645531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:15.645538 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:15.645599 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:15.649560 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.650443 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.650451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.650457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.650616 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.651313 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.651317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.651320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.651696 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:15.651852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:15.655372 (+ 3520us) server_negotiation.cc:299] Negotiation successful
0926 07:27:15.655525 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":253,"threads_started":1}
I20260926 07:27:15.674571 3834 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:42221
I20260926 07:27:15.674973 3969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:42221 every 8 connection(s)
I20260926 07:27:15.675864 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:15.677752 3970 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34681
I20260926 07:27:15.678018 3970 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:15.678792 3970 heartbeater.cc:511] Master 127.2.147.254:34681 requested a full tablet report, sending...
I20260926 07:27:15.679962 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3834
I20260926 07:27:15.680190 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:15.680987 2639 external_mini_cluster.cc:1428] Setting key 70825d99c8b366b4c16a220de37dff0e
I20260926 07:27:15.681052 3763 ts_manager.cc:194] Registered new tserver with Master: 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221)
I20260926 07:27:15.683248 3763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:43943
I20260926 07:27:15.683647 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:15.663647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50729 (local address 127.2.147.254:34681)
0926 07:27:15.663842 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:15.663850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:15.664726 (+ 876us) server_negotiation.cc:407] Connection header received
0926 07:27:15.664950 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:15.664959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:15.665056 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:15.665154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:15.666416 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.667622 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.670323 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.671018 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.671172 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:15.673900 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:15.673907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:15.673912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:15.673945 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:15.677137 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.678210 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.678219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.678225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.678407 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:15.678845 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:15.678853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:15.678859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:15.679318 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:15.679740 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:15.683036 (+ 3296us) server_negotiation.cc:299] Negotiation successful
0926 07:27:15.683309 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:27:15.683670 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:15.663855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.193:50729)
0926 07:27:15.664631 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:15.664646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:15.664789 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:15.665355 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:15.665362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:15.665959 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:15.666273 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:15.666282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.667786 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.667794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:15.670160 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:15.670180 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:15.671372 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:15.671382 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:15.671623 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:15.672505 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:15.672531 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:15.673714 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:15.677276 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:15.677285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:15.677291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:15.678014 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:15.678537 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:15.678550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:15.678553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:15.678716 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:15.679673 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:15.682553 (+ 2880us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:15.682733 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 07:27:15.682908 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 07:27:15.683245 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":464,"threads_started":1}
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:15.718617 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34681
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:16.021755 3974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:16.022068 3974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:16.022228 3974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:16.029313 3974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:16.029462 3974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:16.029603 3974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:16.029735 3974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:16.039546 3974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:16.040800 3974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:16.042119 3974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:16.055269 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:16.055186 3980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:16.055217 3982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:16.056078 3974 server_base.cc:1034] running on GCE node
I20260926 07:27:16.056689 3974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:16.057767 3974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:16.058985 3974 hybrid_clock.cc:648] HybridClock initialized: now 1790407636058934 us; error 33 us; skew 500 ppm
I20260926 07:27:16.059340 3974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:16.062614 3974 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:16.064174 3974 webserver.cc:460] Webserver started at http://127.2.147.194:37133/ using document root <none> and password file <none>
I20260926 07:27:16.064903 3974 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:16.065023 3974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:16.065363 3974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:16.068351 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "e2dbb7b9b69a42fcbd565469021eeef8"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "34233f6749021736d9ae14c9b17f4c22"
server_key_iv: "992c4fa6f8d56e6200c246ab72b42c0a"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.069124 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "e2dbb7b9b69a42fcbd565469021eeef8"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "34233f6749021736d9ae14c9b17f4c22"
server_key_iv: "992c4fa6f8d56e6200c246ab72b42c0a"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.075305 3974 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 07:27:16.079413 3989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:16.080850 3974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:16.080982 3974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "e2dbb7b9b69a42fcbd565469021eeef8"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "34233f6749021736d9ae14c9b17f4c22"
server_key_iv: "992c4fa6f8d56e6200c246ab72b42c0a"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.081179 3974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:16.094331 3974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:16.098479 3974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:16.098724 3974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:16.101153 3974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:16.101265 3974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:16.101392 3974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:16.101478 3974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:16.127723 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.101907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:52859 (local address 127.2.147.254:34681)
0926 07:27:16.102097 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.102103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.103810 (+ 1707us) server_negotiation.cc:407] Connection header received
0926 07:27:16.106935 (+ 3125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.106950 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.107066 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.107157 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:16.109973 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.110888 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.113613 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.114263 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.114366 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.120010 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:16.120019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:16.120024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:16.120067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:16.123298 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.124346 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.124353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.124358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.124525 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.125240 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.125248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.125254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.125685 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:16.125884 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.126834 (+ 950us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.127153 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:27:16.128563 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.102636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.194:52859)
0926 07:27:16.103618 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.103678 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.106506 (+ 2828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.107789 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.107802 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.108771 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:16.109767 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.109796 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.111047 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.111052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.113455 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.113467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.114538 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.114547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.114911 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.115799 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:16.115837 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:16.119707 (+ 3870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:16.123429 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.123438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.123457 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.124177 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.124697 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.124705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.124711 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.125071 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.125854 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:16.125864 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:16.126353 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.126695 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.127192 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":499,"threads_started":1}
I20260926 07:27:16.148676 3974 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:32803
I20260926 07:27:16.148770 4108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:32803 every 8 connection(s)
I20260926 07:27:16.150141 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.134490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55627 (local address 127.2.147.254:34681)
0926 07:27:16.134670 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.134677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.135578 (+ 901us) server_negotiation.cc:407] Connection header received
0926 07:27:16.135802 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.135808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.135887 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.135992 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:16.136836 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.137884 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.140262 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.140818 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.140995 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.143552 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:16.143560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:16.143567 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:16.143609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:16.147338 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.148172 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.148178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.148182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.148328 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.148857 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.148863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.148867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.149198 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:16.149314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.149755 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.149908 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:16.150157 3974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:16.150483 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.134625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.194:55627)
0926 07:27:16.135454 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.135480 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.135631 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.136018 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.136022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.136429 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:16.136701 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.136709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.138043 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.138050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.140114 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.140131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.141058 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.141071 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.141318 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.142048 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:16.142077 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:16.143328 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:16.147476 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.147483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.147488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.148000 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.148494 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.148500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.148509 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.148700 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.149318 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:16.149323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:16.149510 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.149686 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.149975 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":414,"threads_started":1}
I20260926 07:27:16.152220 4109 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34681
I20260926 07:27:16.152458 4109 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:16.153251 4109 heartbeater.cc:511] Master 127.2.147.254:34681 requested a full tablet report, sending...
I20260926 07:27:16.154671 3762 ts_manager.cc:194] Registered new tserver with Master: e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803)
I20260926 07:27:16.154757 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 3974
I20260926 07:27:16.154975 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:16.155473 2639 external_mini_cluster.cc:1428] Setting key 1e09154d63283d1cf3843ee39b556608
I20260926 07:27:16.155705 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:52859
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:16.191951 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34681
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:16.503055 4113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:16.503350 4113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:16.503444 4113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:16.510264 4113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:16.510355 4113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:16.510428 4113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:16.510492 4113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:16.519793 4113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:16.520939 4113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:16.522146 4113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:16.534777 4118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:16.534736 4119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:16.534811 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:16.535686 4113 server_base.cc:1034] running on GCE node
I20260926 07:27:16.536252 4113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:16.537038 4113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:16.538257 4113 hybrid_clock.cc:648] HybridClock initialized: now 1790407636538185 us; error 42 us; skew 500 ppm
I20260926 07:27:16.538501 4113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:16.541431 4113 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:16.542771 4113 webserver.cc:460] Webserver started at http://127.2.147.195:38627/ using document root <none> and password file <none>
I20260926 07:27:16.543516 4113 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:16.543628 4113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:16.543946 4113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:16.546798 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "f306c0179bbb4202af3e24145e63f657"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "e594139f154a4640d035a5018ca55621"
server_key_iv: "eb9462bc34b9beb325f8540438c3ca55"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.547597 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "f306c0179bbb4202af3e24145e63f657"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "e594139f154a4640d035a5018ca55621"
server_key_iv: "eb9462bc34b9beb325f8540438c3ca55"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.552402 4113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:27:16.555694 4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:16.556941 4113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:27:16.557120 4113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "f306c0179bbb4202af3e24145e63f657"
format_stamp: "Formatted at 2026-09-26 07:27:16 on dist-test-slave-1dwd"
server_key: "e594139f154a4640d035a5018ca55621"
server_key_iv: "eb9462bc34b9beb325f8540438c3ca55"
server_key_version: "encryptionkey@0"
I20260926 07:27:16.557327 4113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:16.578119 4113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:16.582522 4113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:16.582787 4113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:16.585254 4113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:16.585366 4113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:16.585494 4113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:16.585588 4113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:16.605551 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.585963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:38325 (local address 127.2.147.254:34681)
0926 07:27:16.586154 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.586160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.587660 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 07:27:16.589807 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.589812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.589913 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.590042 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:16.591908 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.592640 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.594220 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.594640 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.594765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.599521 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:16.599528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:16.599533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:16.599570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:16.602222 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.602911 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.602917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.602920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.603047 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.603513 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.603518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.603522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.604008 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:16.604163 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.604964 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.605178 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:27:16.607026 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.586514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.195:38325)
0926 07:27:16.587465 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.587516 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.589444 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.590272 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.590288 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.591173 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:16.591752 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.591767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.592783 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.592789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.594078 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.594089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.594871 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.594879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.595077 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.595800 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:16.595843 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:16.599195 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:16.602357 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.602366 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.602382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.602784 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.603159 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.603165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.603169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.603391 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.604142 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:16.604153 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:16.604529 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.604866 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.605422 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":421,"threads_started":1}
I20260926 07:27:16.630300 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.613904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.2.147.195:55009)
0926 07:27:16.614579 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.614604 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.614800 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.615115 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.615122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.615934 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:16.616462 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.616478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.617811 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.617818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.620182 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.620197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.620911 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.620919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.621178 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.621875 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:16.621907 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:16.623183 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:16.627016 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.627021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.627025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.627667 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.628131 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:16.628139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:16.628143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:16.628322 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:16.628988 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:16.628993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:16.629180 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.629338 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.629705 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":325,"threads_started":1}
I20260926 07:27:16.631498 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.614282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:55009 (local address 127.2.147.254:34681)
0926 07:27:16.614553 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.614557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.614700 (+ 143us) server_negotiation.cc:407] Connection header received
0926 07:27:16.614915 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.614918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.614973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.615046 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:16.616631 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.617668 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.620334 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.620709 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.620802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.623428 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:16.623436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:16.623442 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:16.623482 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:16.626835 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.627842 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.627849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.627856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.627971 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.628477 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.628482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.628485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.628876 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:16.629062 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.630690 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.630968 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260926 07:27:16.641306 4113 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:41145
I20260926 07:27:16.641350 4247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:41145 every 8 connection(s)
I20260926 07:27:16.642493 4113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:16.644500 4248 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34681
I20260926 07:27:16.644724 4248 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:16.645332 4248 heartbeater.cc:511] Master 127.2.147.254:34681 requested a full tablet report, sending...
I20260926 07:27:16.646557 3762 ts_manager.cc:194] Registered new tserver with Master: f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145)
I20260926 07:27:16.647516 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:38325
I20260926 07:27:16.647702 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 4113
I20260926 07:27:16.647887 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:16.648290 2639 external_mini_cluster.cc:1428] Setting key cfbe39b53f606c6afa1f8f2ba68f7c0b
I20260926 07:27:16.650940 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 07:27:16.687273 3970 heartbeater.cc:503] Master 127.2.147.254:34681 was elected leader, sending a full tablet report...
Sep 26 07:27:16 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:16.694696 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.681473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.2.147.254:34681)
0926 07:27:16.681695 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.681702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.681935 (+ 233us) server_negotiation.cc:407] Connection header received
0926 07:27:16.682148 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.682151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.682230 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.682335 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:16.683722 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.684434 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.685702 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.686181 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.686312 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.690255 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:16.690260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:16.690263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:16.690312 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:16.692653 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.693235 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.693238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.693241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.693345 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:16.693644 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:16.693646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:16.693648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:16.693892 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:16.694058 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.694284 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.694431 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:27:16.698195 3762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:16.704675 3762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:16.728112 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.716305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:32803 (local address 127.0.0.1:60004)
0926 07:27:16.717960 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.717985 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.718087 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.720621 (+ 2534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.720631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.720673 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.721199 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.721225 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.723019 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.723024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.725697 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.725711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.727046 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.727053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.727276 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.727327 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.727488 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.727657 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"mutex_wait_us":92,"thread_start_us":1691,"threads_started":1}
I20260926 07:27:16.728796 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.716303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.2.147.195:41145)
0926 07:27:16.720477 (+ 4174us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.720486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.720518 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:27:16.720608 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.720615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.720829 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.721028 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.721794 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.723162 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.725893 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.727283 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.727399 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.727637 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.728224 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.728437 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4009,"thread_start_us":3846,"threads_started":1}
I20260926 07:27:16.728950 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.717235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.2.147.194:32803)
0926 07:27:16.719678 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.719971 (+ 293us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.720004 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:27:16.720110 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.720119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.720381 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.720569 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.721491 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.722909 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.725860 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.726857 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.727009 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.727313 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.728519 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.728687 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":2120,"threads_started":1}
I20260926 07:27:16.729183 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.715989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41145 (local address 127.0.0.1:49476)
0926 07:27:16.717094 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.717147 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.717318 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.721021 (+ 3703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.721025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.721050 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.721565 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.721582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.723296 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.723302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.725697 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.725711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.727459 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.727466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.727630 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.727657 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.727781 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.727903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":82,"thread_start_us":5422,"threads_started":1}
I20260926 07:27:16.730355 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.716530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:42221 (local address 127.0.0.1:55242)
0926 07:27:16.721718 (+ 5188us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.721741 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.721841 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.722455 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.722459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.722482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.722790 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.722798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.724154 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.724162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.726086 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.726101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.729566 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.729575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.729752 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.729781 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.729902 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.730024 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5051,"mutex_wait_us":1,"thread_start_us":1145,"threads_started":1}
I20260926 07:27:16.731268 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.716874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.2.147.193:42221)
0926 07:27:16.720874 (+ 4000us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.720882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.721958 (+ 1076us) server_negotiation.cc:407] Connection header received
0926 07:27:16.722063 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.722071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.722279 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.722455 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.722958 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.724021 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.726245 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.727731 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.727866 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.728053 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.730797 (+ 2744us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.730955 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3865,"thread_start_us":2950,"threads_started":1}
I20260926 07:27:16.732815 4039 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 (DEFAULT_TABLE table=test-table [id=9476207ef6b74515a0e7f83e29140e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.732842 4180 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 (DEFAULT_TABLE table=test-table [id=9476207ef6b74515a0e7f83e29140e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.734028 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa9a5610dd04e798e7549cd9a5c1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.734222 4180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa9a5610dd04e798e7549cd9a5c1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.734524 3900 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 (DEFAULT_TABLE table=test-table [id=9476207ef6b74515a0e7f83e29140e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.735728 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa9a5610dd04e798e7549cd9a5c1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.744994 4267 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Bootstrap starting.
I20260926 07:27:16.748119 4267 tablet_bootstrap.cc:654] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.748493 4268 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Bootstrap starting.
I20260926 07:27:16.750317 4267 log.cc:824] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:16.752249 4268 tablet_bootstrap.cc:654] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.752408 4269 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Bootstrap starting.
I20260926 07:27:16.753409 4268 log.cc:824] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:16.753526 4267 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: No bootstrap required, opened a new log
I20260926 07:27:16.753764 4267 ts_tablet_manager.cc:1397] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:27:16.754956 4269 tablet_bootstrap.cc:654] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.756680 4269 log.cc:824] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:16.758227 4268 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: No bootstrap required, opened a new log
I20260926 07:27:16.758456 4268 ts_tablet_manager.cc:1397] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260926 07:27:16.759451 4269 tablet_bootstrap.cc:492] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: No bootstrap required, opened a new log
I20260926 07:27:16.759711 4269 ts_tablet_manager.cc:1397] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260926 07:27:16.764269 4267 raft_consensus.cc:348] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.764681 4267 raft_consensus.cc:374] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.764775 4267 raft_consensus.cc:729] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dbb7b9b69a42fcbd565469021eeef8, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.765364 4267 consensus_queue.cc:260] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.766422 4109 heartbeater.cc:503] Master 127.2.147.254:34681 was elected leader, sending a full tablet report...
I20260926 07:27:16.767014 4267 ts_tablet_manager.cc:1428] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 07:27:16.767335 4269 raft_consensus.cc:348] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.767735 4269 raft_consensus.cc:374] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.767834 4269 raft_consensus.cc:729] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec16586d401422fa8fdfa7acc81f2de, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.768364 4269 consensus_queue.cc:260] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.770969 4269 ts_tablet_manager.cc:1428] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260926 07:27:16.770769 4268 raft_consensus.cc:348] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.771136 4268 raft_consensus.cc:374] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.771266 4268 raft_consensus.cc:729] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f306c0179bbb4202af3e24145e63f657, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.771801 4268 consensus_queue.cc:260] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.773478 4248 heartbeater.cc:503] Master 127.2.147.254:34681 was elected leader, sending a full tablet report...
I20260926 07:27:16.774081 4268 ts_tablet_manager.cc:1428] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260926 07:27:16.803797 4274 raft_consensus.cc:482] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:16.804041 4274 raft_consensus.cc:504] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.805657 4274 leader_election.cc:288] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145), e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803)
I20260926 07:27:16.812872 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.806049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:59653 (local address 127.2.147.195:41145)
0926 07:27:16.806247 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.806256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.806889 (+ 633us) server_negotiation.cc:407] Connection header received
0926 07:27:16.807095 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.807106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.807157 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.807306 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.807816 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.808910 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.810985 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.811824 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.811998 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.812070 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.812499 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.812619 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:27:16.812978 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.805851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41145 (local address 127.2.147.193:59653)
0926 07:27:16.806776 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.806794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.806960 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.807320 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.807324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.807345 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.807686 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.807695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.809072 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.809077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.810844 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.810855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.812061 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.812068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.812275 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.812325 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.812418 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.812569 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":542,"threads_started":1}
I20260926 07:27:16.814568 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.806122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:32803 (local address 127.2.147.193:52077)
0926 07:27:16.806991 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.807004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.807140 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.807464 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.807468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.807513 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.807915 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.807928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.809278 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.809286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.811785 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.811801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.813752 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.813758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.813925 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.813966 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.814064 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.814183 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"spinlock_wait_cycles":3456,"thread_start_us":595,"threads_started":1}
I20260926 07:27:16.814354 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa9a5610dd04e798e7549cd9a5c1e33" candidate_uuid: "5ec16586d401422fa8fdfa7acc81f2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f306c0179bbb4202af3e24145e63f657" is_pre_election: true
I20260926 07:27:16.814579 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.806095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:52077 (local address 127.2.147.194:32803)
0926 07:27:16.806289 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.806298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.807102 (+ 804us) server_negotiation.cc:407] Connection header received
0926 07:27:16.807290 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.807295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.807357 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.807446 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.808088 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.809126 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.811955 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.813549 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.813728 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.813797 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.814116 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.814285 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:27:16.814937 4203 raft_consensus.cc:2454] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec16586d401422fa8fdfa7acc81f2de in term 0.
I20260926 07:27:16.815618 3851 leader_election.cc:302] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec16586d401422fa8fdfa7acc81f2de, f306c0179bbb4202af3e24145e63f657; no voters:
I20260926 07:27:16.815657 4059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa9a5610dd04e798e7549cd9a5c1e33" candidate_uuid: "5ec16586d401422fa8fdfa7acc81f2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" is_pre_election: true
I20260926 07:27:16.815948 4274 raft_consensus.cc:2781] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:16.816082 4274 raft_consensus.cc:482] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:16.816167 4274 raft_consensus.cc:3037] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:16.816164 4059 raft_consensus.cc:2454] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ec16586d401422fa8fdfa7acc81f2de in term 0.
I20260926 07:27:16.818576 4274 raft_consensus.cc:504] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.819242 4274 leader_election.cc:288] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [CANDIDATE]: Term 1 election: Requested vote from peers f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145), e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803)
I20260926 07:27:16.819584 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa9a5610dd04e798e7549cd9a5c1e33" candidate_uuid: "5ec16586d401422fa8fdfa7acc81f2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f306c0179bbb4202af3e24145e63f657"
I20260926 07:27:16.819778 4203 raft_consensus.cc:3037] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:16.819797 4059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa9a5610dd04e798e7549cd9a5c1e33" candidate_uuid: "5ec16586d401422fa8fdfa7acc81f2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dbb7b9b69a42fcbd565469021eeef8"
I20260926 07:27:16.819954 4059 raft_consensus.cc:3037] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:16.821769 4203 raft_consensus.cc:2454] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec16586d401422fa8fdfa7acc81f2de in term 1.
I20260926 07:27:16.822052 4059 raft_consensus.cc:2454] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec16586d401422fa8fdfa7acc81f2de in term 1.
I20260926 07:27:16.822096 3851 leader_election.cc:302] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec16586d401422fa8fdfa7acc81f2de, f306c0179bbb4202af3e24145e63f657; no voters:
I20260926 07:27:16.822371 4274 raft_consensus.cc:2781] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:16.822763 4274 raft_consensus.cc:686] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 1 LEADER]: Becoming Leader. State: Replica: 5ec16586d401422fa8fdfa7acc81f2de, State: Running, Role: LEADER
I20260926 07:27:16.823191 4274 consensus_queue.cc:237] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } }
I20260926 07:27:16.827968 3763 catalog_manager.cc:5495] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193). New cstate: current_term: 1 leader_uuid: "5ec16586d401422fa8fdfa7acc81f2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:16.857208 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.852357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.2.147.193:42221)
0926 07:27:16.852533 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.852537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.852572 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:27:16.852851 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.852855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.852935 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.853094 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:16.853431 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.854138 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.855421 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.855787 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.855873 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.856260 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:16.856739 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:16.856845 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.856952 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.857010 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:16.865067 4059 raft_consensus.cc:1261] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Refusing update from remote peer 5ec16586d401422fa8fdfa7acc81f2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:16.865039 4203 raft_consensus.cc:1261] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Refusing update from remote peer 5ec16586d401422fa8fdfa7acc81f2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:16.865999 4278 consensus_queue.cc:1035] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:16.866312 4274 consensus_queue.cc:1035] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:16.879896 4283 mvcc.cc:205] Tried to move back new op lower bound from 7333509680589971456 to 7333509680431300608. Current Snapshot: MvccSnapshot[applied={T|T < 7333509680589971456}]
I20260926 07:27:16.896754 3762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:16.897456 3762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:16.900970 3762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Sending DeleteTablet for 3 replicas of tablet ffa9a5610dd04e798e7549cd9a5c1e33
I20260926 07:27:16.902022 4180 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:16 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:49476
I20260926 07:27:16.902040 3900 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:16 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:55242
I20260926 07:27:16.902040 4039 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9a5610dd04e798e7549cd9a5c1e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:16 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:60004
I20260926 07:27:16.905431 3762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:16.906056 3762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:16.914770 4291 tablet_replica.cc:330] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: stopping tablet replica
I20260926 07:27:16.914947 4293 tablet_replica.cc:330] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: stopping tablet replica
I20260926 07:27:16.915947 4178 tablet_service.cc:1467] Processing CreateTablet for tablet e32d8a7fb48741ebb7795febc892f81d (DEFAULT_TABLE table=test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.915980 3899 tablet_service.cc:1467] Processing CreateTablet for tablet e32d8a7fb48741ebb7795febc892f81d (DEFAULT_TABLE table=test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.916486 3899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d8a7fb48741ebb7795febc892f81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.916482 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d8a7fb48741ebb7795febc892f81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.917080 4039 tablet_service.cc:1467] Processing CreateTablet for tablet e32d8a7fb48741ebb7795febc892f81d (DEFAULT_TABLE table=test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:16.917507 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d8a7fb48741ebb7795febc892f81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:16.923043 4268 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Bootstrap starting.
I20260926 07:27:16.923137 4269 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Bootstrap starting.
I20260926 07:27:16.925139 4268 tablet_bootstrap.cc:654] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.925133 4269 tablet_bootstrap.cc:654] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.926988 4267 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Bootstrap starting.
I20260926 07:27:16.928658 4293 raft_consensus.cc:2229] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:16.928979 4267 tablet_bootstrap.cc:654] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:16.929239 4293 raft_consensus.cc:2258] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:16.932439 4293 ts_tablet_manager.cc:1905] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:16.936029 4269 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: No bootstrap required, opened a new log
I20260926 07:27:16.936190 4269 ts_tablet_manager.cc:1397] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260926 07:27:16.937192 4292 tablet_replica.cc:330] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: stopping tablet replica
I20260926 07:27:16.937319 4269 raft_consensus.cc:348] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.937566 4269 raft_consensus.cc:374] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.937678 4269 raft_consensus.cc:729] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec16586d401422fa8fdfa7acc81f2de, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.937963 4269 consensus_queue.cc:260] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.938390 4292 raft_consensus.cc:2229] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:16.938556 4269 ts_tablet_manager.cc:1428] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:27:16.939137 4292 raft_consensus.cc:2258] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:16.941151 4292 ts_tablet_manager.cc:1905] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:16.941367 4293 ts_tablet_manager.cc:1918] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:16.941504 4293 log.cc:1196] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/ffa9a5610dd04e798e7549cd9a5c1e33
I20260926 07:27:16.942886 4293 ts_tablet_manager.cc:1939] T ffa9a5610dd04e798e7549cd9a5c1e33 P f306c0179bbb4202af3e24145e63f657: Deleting consensus metadata
I20260926 07:27:16.943363 4268 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: No bootstrap required, opened a new log
I20260926 07:27:16.943570 4268 ts_tablet_manager.cc:1397] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20260926 07:27:16.944759 3748 catalog_manager.cc:4841] TS f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145): tablet ffa9a5610dd04e798e7549cd9a5c1e33 (table test-table [id=9476207ef6b74515a0e7f83e29140e0d]) successfully deleted
I20260926 07:27:16.945568 4268 raft_consensus.cc:348] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.945801 4268 raft_consensus.cc:374] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.945912 4268 raft_consensus.cc:729] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f306c0179bbb4202af3e24145e63f657, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.946296 4267 tablet_bootstrap.cc:492] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: No bootstrap required, opened a new log
I20260926 07:27:16.946209 4268 consensus_queue.cc:260] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.946787 4268 ts_tablet_manager.cc:1428] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:16.947023 4267 ts_tablet_manager.cc:1397] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20260926 07:27:16.947867 4267 raft_consensus.cc:348] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.948103 4267 raft_consensus.cc:374] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:16.948215 4267 raft_consensus.cc:729] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dbb7b9b69a42fcbd565469021eeef8, State: Initialized, Role: FOLLOWER
I20260926 07:27:16.948464 4267 consensus_queue.cc:260] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.949028 4267 ts_tablet_manager.cc:1428] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:27:16.950480 4291 raft_consensus.cc:2229] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:16.950764 4292 ts_tablet_manager.cc:1918] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:16.950922 4292 log.cc:1196] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/ffa9a5610dd04e798e7549cd9a5c1e33
I20260926 07:27:16.950963 4291 raft_consensus.cc:2258] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:16.951496 4292 ts_tablet_manager.cc:1939] T ffa9a5610dd04e798e7549cd9a5c1e33 P 5ec16586d401422fa8fdfa7acc81f2de: Deleting consensus metadata
I20260926 07:27:16.953104 3747 catalog_manager.cc:4841] TS 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221): tablet ffa9a5610dd04e798e7549cd9a5c1e33 (table test-table [id=9476207ef6b74515a0e7f83e29140e0d]) successfully deleted
I20260926 07:27:16.953672 4291 ts_tablet_manager.cc:1905] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:16.958352 4291 ts_tablet_manager.cc:1918] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:16.958534 4291 log.cc:1196] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/ffa9a5610dd04e798e7549cd9a5c1e33
I20260926 07:27:16.959049 4291 ts_tablet_manager.cc:1939] T ffa9a5610dd04e798e7549cd9a5c1e33 P e2dbb7b9b69a42fcbd565469021eeef8: Deleting consensus metadata
I20260926 07:27:16.960638 3750 catalog_manager.cc:4841] TS e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803): tablet ffa9a5610dd04e798e7549cd9a5c1e33 (table test-table [id=9476207ef6b74515a0e7f83e29140e0d]) successfully deleted
I20260926 07:27:16.983703 4275 raft_consensus.cc:482] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:16.983928 4275 raft_consensus.cc:504] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.985426 4275 leader_election.cc:288] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221), e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803)
I20260926 07:27:16.992856 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.986081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43999 (local address 127.2.147.194:32803)
0926 07:27:16.986608 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.986614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.986632 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:16.986844 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.986850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.986922 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.986985 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.987871 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.989008 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.991170 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.991923 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.992100 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.992172 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.992440 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.992575 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260926 07:27:16.993095 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.985651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:42221 (local address 127.2.147.195:33621)
0926 07:27:16.985891 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.985917 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.986032 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.987276 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.987281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.987343 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.987623 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.987635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.988691 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.988696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.990514 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.990528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.992255 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.992261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.992447 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.992479 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.992593 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.992728 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":2,"spinlock_wait_cycles":32000}
I20260926 07:27:16.993345 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.985830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:32803 (local address 127.2.147.195:43999)
0926 07:27:16.986528 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:16.986543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:16.986635 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:16.987180 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:16.987188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:16.987211 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:16.987675 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.987685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.989138 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.989142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.991036 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:16.991046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.992128 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.992134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:16.992285 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.992308 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:27:16.992416 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:27:16.992559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":41,"thread_start_us":461,"threads_started":1}
I20260926 07:27:16.993924 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:16.986648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:33621 (local address 127.2.147.193:42221)
0926 07:27:16.986936 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:16.986947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:16.986970 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:27:16.987031 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:16.987036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:16.987105 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:16.987221 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:16.987796 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.988557 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.990655 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:16.991957 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:16.992105 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:16.992174 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:16.993154 (+ 980us) server_negotiation.cc:299] Negotiation successful
0926 07:27:16.993305 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260926 07:27:16.993976 4059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d8a7fb48741ebb7795febc892f81d" candidate_uuid: "f306c0179bbb4202af3e24145e63f657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" is_pre_election: true
I20260926 07:27:16.994242 4059 raft_consensus.cc:2454] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f306c0179bbb4202af3e24145e63f657 in term 0.
I20260926 07:27:16.994712 4132 leader_election.cc:302] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2dbb7b9b69a42fcbd565469021eeef8, f306c0179bbb4202af3e24145e63f657; no voters:
I20260926 07:27:16.994737 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d8a7fb48741ebb7795febc892f81d" candidate_uuid: "f306c0179bbb4202af3e24145e63f657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec16586d401422fa8fdfa7acc81f2de" is_pre_election: true
I20260926 07:27:16.995045 4275 raft_consensus.cc:2781] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:16.995170 4275 raft_consensus.cc:482] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:16.995296 4275 raft_consensus.cc:3037] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:16.995177 3920 raft_consensus.cc:2454] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f306c0179bbb4202af3e24145e63f657 in term 0.
I20260926 07:27:16.998302 4275 raft_consensus.cc:504] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:16.998960 4275 leader_election.cc:288] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221), e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803)
I20260926 07:27:16.999444 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d8a7fb48741ebb7795febc892f81d" candidate_uuid: "f306c0179bbb4202af3e24145e63f657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec16586d401422fa8fdfa7acc81f2de"
I20260926 07:27:16.999540 4059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d8a7fb48741ebb7795febc892f81d" candidate_uuid: "f306c0179bbb4202af3e24145e63f657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dbb7b9b69a42fcbd565469021eeef8"
I20260926 07:27:16.999647 3920 raft_consensus.cc:3037] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:16.999794 4059 raft_consensus.cc:3037] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:17.002058 3920 raft_consensus.cc:2454] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f306c0179bbb4202af3e24145e63f657 in term 1.
I20260926 07:27:17.002451 4129 leader_election.cc:302] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec16586d401422fa8fdfa7acc81f2de, f306c0179bbb4202af3e24145e63f657; no voters:
I20260926 07:27:17.002724 4275 raft_consensus.cc:2781] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:17.002722 4059 raft_consensus.cc:2454] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f306c0179bbb4202af3e24145e63f657 in term 1.
I20260926 07:27:17.003315 4275 raft_consensus.cc:686] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 1 LEADER]: Becoming Leader. State: Replica: f306c0179bbb4202af3e24145e63f657, State: Running, Role: LEADER
I20260926 07:27:17.003702 4275 consensus_queue.cc:237] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.007323 3761 catalog_manager.cc:5495] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 reported cstate change: term changed from 0 to 1, leader changed from <none> to f306c0179bbb4202af3e24145e63f657 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "f306c0179bbb4202af3e24145e63f657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } health_report { overall_health: HEALTHY } } }
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407635, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:17.055789 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.041454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.2.147.254:34681)
0926 07:27:17.041607 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.041612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.042180 (+ 568us) server_negotiation.cc:407] Connection header received
0926 07:27:17.042325 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.042331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.042422 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.042535 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.043654 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.044776 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.046052 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.047147 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.047309 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.050613 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.050619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.050623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.050661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.053151 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.053806 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.053811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.053815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.053927 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.054268 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.054275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.054280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.054579 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.054708 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.055202 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.055518 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":51}
I20260926 07:27:17.055904 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.041555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34681 (local address 127.0.0.1:55500)
0926 07:27:17.042077 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.042094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.042208 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.042567 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.042570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.043185 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:17.043507 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.043516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.045015 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.045019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.045921 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.045930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.047353 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.047360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.047549 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.048220 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:17.048249 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:17.050454 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:17.053291 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:17.053301 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:17.053308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:17.053678 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:17.054029 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:17.054036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:17.054039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:17.054163 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:17.054724 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:17.054733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:17.054930 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.055111 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.055400 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":300,"threads_started":1}
I20260926 07:27:17.061163 3761 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:55500:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:27:17.062235 3761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:17.068650 4039 tablet_service.cc:1467] Processing CreateTablet for tablet ee1632bfebb14ecaa316d7e88c5a1dfb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e33fb7b945940f395fef12040241b69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:17.068641 3900 tablet_service.cc:1467] Processing CreateTablet for tablet ee1632bfebb14ecaa316d7e88c5a1dfb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e33fb7b945940f395fef12040241b69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:17.069252 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1632bfebb14ecaa316d7e88c5a1dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:17.069259 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1632bfebb14ecaa316d7e88c5a1dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:17.071698 4178 tablet_service.cc:1467] Processing CreateTablet for tablet ee1632bfebb14ecaa316d7e88c5a1dfb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e33fb7b945940f395fef12040241b69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:17.072170 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1632bfebb14ecaa316d7e88c5a1dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:17.076576 4269 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: Bootstrap starting.
I20260926 07:27:17.078234 4269 tablet_bootstrap.cc:654] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:17.086256 4268 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: Bootstrap starting.
I20260926 07:27:17.088385 4268 tablet_bootstrap.cc:654] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:17.092484 4267 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: Bootstrap starting.
I20260926 07:27:17.095067 4267 tablet_bootstrap.cc:654] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:17.099435 4267 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: No bootstrap required, opened a new log
I20260926 07:27:17.099630 4267 ts_tablet_manager.cc:1397] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:27:17.100468 4267 raft_consensus.cc:348] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.100725 4267 raft_consensus.cc:374] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:17.100852 4267 raft_consensus.cc:729] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dbb7b9b69a42fcbd565469021eeef8, State: Initialized, Role: FOLLOWER
I20260926 07:27:17.101219 4268 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: No bootstrap required, opened a new log
I20260926 07:27:17.101147 4267 consensus_queue.cc:260] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.101387 4268 ts_tablet_manager.cc:1397] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260926 07:27:17.101588 4273 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.101909 4273 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.102089 4268 raft_consensus.cc:348] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.102316 4268 raft_consensus.cc:374] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:17.102378 4268 raft_consensus.cc:729] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f306c0179bbb4202af3e24145e63f657, State: Initialized, Role: FOLLOWER
I20260926 07:27:17.102632 4268 consensus_queue.cc:260] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.103166 4267 ts_tablet_manager.cc:1428] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:17.104610 4269 tablet_bootstrap.cc:492] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: No bootstrap required, opened a new log
I20260926 07:27:17.104796 4269 ts_tablet_manager.cc:1397] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.010s
I20260926 07:27:17.105542 4269 raft_consensus.cc:348] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.105779 4269 raft_consensus.cc:374] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:17.105875 4269 raft_consensus.cc:729] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec16586d401422fa8fdfa7acc81f2de, State: Initialized, Role: FOLLOWER
I20260926 07:27:17.106150 4269 consensus_queue.cc:260] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.106571 4286 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.106945 4286 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.108258 4269 ts_tablet_manager.cc:1428] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:27:17.108776 4275 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.109071 4275 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.110085 4268 ts_tablet_manager.cc:1428] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
W20260926 07:27:17.145205 4249 tablet.cc:2369] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:17.153751 4110 tablet.cc:2369] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:17.188372 3971 tablet.cc:2369] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:17.200822 4273 raft_consensus.cc:482] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:17.201074 4273 raft_consensus.cc:504] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.202629 4273 leader_election.cc:288] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221), f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145)
I20260926 07:27:17.210887 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.203473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:50509 (local address 127.2.147.195:41145)
0926 07:27:17.203669 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.203675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.204256 (+ 581us) server_negotiation.cc:407] Connection header received
0926 07:27:17.204460 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.204464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.204535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.204621 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:17.205756 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.206781 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.208846 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.209893 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.210051 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.210148 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.210481 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.210620 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:27:17.211000 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.202976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41145 (local address 127.2.147.194:50509)
0926 07:27:17.204167 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.204184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.204326 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.205165 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.205175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.205199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:17.205595 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.205608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.206902 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.206907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.208704 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.208716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.210125 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.210132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.210308 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.210343 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.210455 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.210594 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":95,"thread_start_us":898,"threads_started":1}
I20260926 07:27:17.212054 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.203031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58649 (local address 127.2.147.193:42221)
0926 07:27:17.203328 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.203336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.203964 (+ 628us) server_negotiation.cc:407] Connection header received
0926 07:27:17.204262 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.204268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.204347 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.204451 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:17.205503 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.206705 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.209476 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.210893 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.211067 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.211130 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.211654 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.211785 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:27:17.212116 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.202836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:42221 (local address 127.2.147.194:58649)
0926 07:27:17.203835 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.203861 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.204080 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.204476 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.204481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.204500 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:17.205274 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.205292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.206846 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.206851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.209306 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.209322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.211306 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.211313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.211481 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.211505 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.211627 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.211750 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":88,"thread_start_us":711,"threads_started":1}
I20260926 07:27:17.212425 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1632bfebb14ecaa316d7e88c5a1dfb" candidate_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f306c0179bbb4202af3e24145e63f657" is_pre_election: true
I20260926 07:27:17.212657 4203 raft_consensus.cc:2454] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dbb7b9b69a42fcbd565469021eeef8 in term 0.
I20260926 07:27:17.213183 3991 leader_election.cc:302] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2dbb7b9b69a42fcbd565469021eeef8, f306c0179bbb4202af3e24145e63f657; no voters:
I20260926 07:27:17.213279 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1632bfebb14ecaa316d7e88c5a1dfb" candidate_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec16586d401422fa8fdfa7acc81f2de" is_pre_election: true
I20260926 07:27:17.213519 3920 raft_consensus.cc:2454] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dbb7b9b69a42fcbd565469021eeef8 in term 0.
I20260926 07:27:17.213815 4273 raft_consensus.cc:2781] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:17.213948 4273 raft_consensus.cc:482] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:17.214051 4273 raft_consensus.cc:3037] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:17.216215 4273 raft_consensus.cc:504] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.216799 4273 leader_election.cc:288] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221), f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145)
I20260926 07:27:17.217244 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1632bfebb14ecaa316d7e88c5a1dfb" candidate_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec16586d401422fa8fdfa7acc81f2de"
I20260926 07:27:17.217464 3920 raft_consensus.cc:3037] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:17.220034 3920 raft_consensus.cc:2454] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dbb7b9b69a42fcbd565469021eeef8 in term 1.
I20260926 07:27:17.220525 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1632bfebb14ecaa316d7e88c5a1dfb" candidate_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f306c0179bbb4202af3e24145e63f657"
I20260926 07:27:17.220563 3990 leader_election.cc:302] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec16586d401422fa8fdfa7acc81f2de, e2dbb7b9b69a42fcbd565469021eeef8; no voters:
I20260926 07:27:17.220719 4203 raft_consensus.cc:3037] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:17.220862 4273 raft_consensus.cc:2781] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:17.223358 4273 raft_consensus.cc:686] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 LEADER]: Becoming Leader. State: Replica: e2dbb7b9b69a42fcbd565469021eeef8, State: Running, Role: LEADER
I20260926 07:27:17.224084 4273 consensus_queue.cc:237] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } }
I20260926 07:27:17.224448 4203 raft_consensus.cc:2454] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dbb7b9b69a42fcbd565469021eeef8 in term 1.
I20260926 07:27:17.226580 4311 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: TxnStatusTablet state changed. Reason: New leader e2dbb7b9b69a42fcbd565469021eeef8. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.226951 4311 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:17.228112 3761 catalog_manager.cc:5495] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } health_report { overall_health: UNKNOWN } } }
I20260926 07:27:17.230798 4313 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:27:17.231009 4313 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:27:17.252002 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.244202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.2.147.194:32803)
0926 07:27:17.244382 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.244388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.244408 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:27:17.244713 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.244724 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.244848 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.244983 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:17.245536 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.247159 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.249155 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.249848 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.249980 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.250522 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.251265 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.251420 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.251583 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.251675 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:17.252014 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.243830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:32803 (local address 127.0.0.1:60020)
0926 07:27:17.244268 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.244291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.244431 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.245156 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.245161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.245174 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:17.245357 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.245369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.247343 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.247349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.248982 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.248997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.250018 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.250024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.250213 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.250284 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.251419 (+ 1135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.251440 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.251536 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.251706 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260926 07:27:17.256632 4203 raft_consensus.cc:1261] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Refusing update from remote peer e2dbb7b9b69a42fcbd565469021eeef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:17.256978 3920 raft_consensus.cc:1261] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Refusing update from remote peer e2dbb7b9b69a42fcbd565469021eeef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:17.257532 4311 consensus_queue.cc:1035] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:17.257824 4273 consensus_queue.cc:1035] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:17.272804 4286 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: TxnStatusTablet state changed. Reason: New leader e2dbb7b9b69a42fcbd565469021eeef8. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.273259 4286 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.275628 4311 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.275888 4311 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:17.276065 4282 mvcc.cc:205] Tried to move back new op lower bound from 7333509682195927040 to 7333509682074263552. Current Snapshot: MvccSnapshot[applied={T|T < 7333509682195927040 or (T in {7333509682195927040})}]
I20260926 07:27:17.276175 4275 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: TxnStatusTablet state changed. Reason: New leader e2dbb7b9b69a42fcbd565469021eeef8. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.276414 4275 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.277036 4281 mvcc.cc:205] Tried to move back new op lower bound from 7333509682195927040 to 7333509682074263552. Current Snapshot: MvccSnapshot[applied={T|T < 7333509682195927040 or (T in {7333509682195927040})}]
I20260926 07:27:17.277429 4286 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.277660 4286 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P 5ec16586d401422fa8fdfa7acc81f2de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.284091 4273 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.284360 4273 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P e2dbb7b9b69a42fcbd565469021eeef8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:17.285590 4275 tablet_replica.cc:439] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 } } peers { permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 } } peers { permanent_uuid: "f306c0179bbb4202af3e24145e63f657" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41145 } } }
I20260926 07:27:17.285941 4275 tablet_replica.cc:442] T ee1632bfebb14ecaa316d7e88c5a1dfb P f306c0179bbb4202af3e24145e63f657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:17.292927 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.285636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.2.147.195:41145)
0926 07:27:17.285855 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.285861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.285879 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:17.285970 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.285975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.286096 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.286236 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:17.286841 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.287955 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.289874 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.290574 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.290696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.291366 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.292090 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.292244 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.292381 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.292462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 07:27:17.314977 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.306784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:35067 (local address 127.2.147.194:32803)
0926 07:27:17.306995 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.307002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.308062 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 07:27:17.308259 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.308264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.308358 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.308467 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:17.309054 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.310251 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.312262 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.313098 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.313245 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.313797 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.313940 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.314437 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.314574 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.314673 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:27:17.315330 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.306823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:32803 (local address 127.2.147.195:35067)
0926 07:27:17.307940 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.307960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.308108 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.308498 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.308539 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.308559 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:17.308864 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.308876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.310438 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.310443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.312101 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.312117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.313289 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.313297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.313512 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.313603 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.314449 (+ 846us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.314461 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.314556 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.314710 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":800,"threads_started":1}
I20260926 07:27:17.326894 3920 raft_consensus.cc:1261] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Refusing update from remote peer f306c0179bbb4202af3e24145e63f657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:17.326960 4059 raft_consensus.cc:1261] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Refusing update from remote peer f306c0179bbb4202af3e24145e63f657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:17.327639 4298 consensus_queue.cc:1035] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec16586d401422fa8fdfa7acc81f2de" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:17.328336 4298 consensus_queue.cc:1035] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dbb7b9b69a42fcbd565469021eeef8" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:17.404982 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.384427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:47017 (local address 127.2.147.195:41145)
0926 07:27:17.384601 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.384605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.397966 (+ 13361us) server_negotiation.cc:407] Connection header received
0926 07:27:17.398157 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.398163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.398252 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.398354 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:17.398998 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.400557 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.402762 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.403492 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.403623 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.404133 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.404270 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.404356 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.404605 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.404690 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:27:17.405141 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.384069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41145 (local address 127.2.147.194:47017)
0926 07:27:17.397837 (+ 13768us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:17.397858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:17.398010 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:17.398380 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:17.398386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:17.398398 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:17.398811 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.398824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.400727 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.400734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.402605 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:17.402621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.403683 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.403690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:17.403868 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.403944 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:17.404411 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:17.404424 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:27:17.404540 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:27:17.404690 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13555,"thread_start_us":13390,"threads_started":1}
I20260926 07:27:17.449056 3762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:17.449317 3762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:17.452608 3762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e3ad818fd2049c5bfeb77cbf9450698: Sending DeleteTablet for 3 replicas of tablet e32d8a7fb48741ebb7795febc892f81d
I20260926 07:27:17.453387 3900 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d8a7fb48741ebb7795febc892f81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:55242
I20260926 07:27:17.453557 4039 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d8a7fb48741ebb7795febc892f81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:60004
I20260926 07:27:17.453830 4291 tablet_replica.cc:330] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: stopping tablet replica
I20260926 07:27:17.454023 4291 raft_consensus.cc:2229] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:17.454241 4291 raft_consensus.cc:2258] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:17.455322 4291 ts_tablet_manager.cc:1905] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:17.458259 4178 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d8a7fb48741ebb7795febc892f81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:49476
I20260926 07:27:17.465976 4330 tablet_replica.cc:330] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: stopping tablet replica
I20260926 07:27:17.466290 4330 raft_consensus.cc:2229] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:17.466547 4330 raft_consensus.cc:2258] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:17.467649 4330 ts_tablet_manager.cc:1905] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:17.472189 4291 ts_tablet_manager.cc:1918] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:17.472342 4291 log.cc:1196] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/e32d8a7fb48741ebb7795febc892f81d
I20260926 07:27:17.472767 4291 ts_tablet_manager.cc:1939] T e32d8a7fb48741ebb7795febc892f81d P e2dbb7b9b69a42fcbd565469021eeef8: Deleting consensus metadata
I20260926 07:27:17.473868 3750 catalog_manager.cc:4841] TS e2dbb7b9b69a42fcbd565469021eeef8 (127.2.147.194:32803): tablet e32d8a7fb48741ebb7795febc892f81d (table test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]) successfully deleted
I20260926 07:27:17.474542 4330 ts_tablet_manager.cc:1918] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:17.474730 4330 log.cc:1196] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/e32d8a7fb48741ebb7795febc892f81d
I20260926 07:27:17.475193 4330 ts_tablet_manager.cc:1939] T e32d8a7fb48741ebb7795febc892f81d P 5ec16586d401422fa8fdfa7acc81f2de: Deleting consensus metadata
I20260926 07:27:17.476202 3747 catalog_manager.cc:4841] TS 5ec16586d401422fa8fdfa7acc81f2de (127.2.147.193:42221): tablet e32d8a7fb48741ebb7795febc892f81d (table test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]) successfully deleted
I20260926 07:27:17.480059 4331 tablet_replica.cc:330] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: stopping tablet replica
I20260926 07:27:17.480314 4331 raft_consensus.cc:2229] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:17.480924 4331 raft_consensus.cc:2258] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:17.481863 4331 ts_tablet_manager.cc:1905] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:17.489213 4331 ts_tablet_manager.cc:1918] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:17.489398 4331 log.cc:1196] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/e32d8a7fb48741ebb7795febc892f81d
I20260926 07:27:17.489971 4331 ts_tablet_manager.cc:1939] T e32d8a7fb48741ebb7795febc892f81d P f306c0179bbb4202af3e24145e63f657: Deleting consensus metadata
I20260926 07:27:17.491284 3748 catalog_manager.cc:4841] TS f306c0179bbb4202af3e24145e63f657 (127.2.147.195:41145): tablet e32d8a7fb48741ebb7795febc892f81d (table test-table [id=ebd7b92f9c8b4a519cfda23bfdf2b48c]) successfully deleted
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:17.505543 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.480367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.2.147.193:42221)
0926 07:27:17.480536 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.480541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.485261 (+ 4720us) server_negotiation.cc:407] Connection header received
0926 07:27:17.485774 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.485780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.485872 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.485940 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.488492 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.489541 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.492009 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.492703 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.492841 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.498244 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.498260 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.498279 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.498318 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.502264 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.503089 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.503100 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.503105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.503310 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.503720 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.503726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.503729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.504163 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.504475 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.504783 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.505003 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:27:17.506245 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260
I20260926 07:27:17.527361 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.509751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.2.147.254:34681)
0926 07:27:17.509961 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.509968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.511704 (+ 1736us) server_negotiation.cc:407] Connection header received
0926 07:27:17.511854 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.511860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.511940 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.512043 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.513631 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.514790 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.517100 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.517776 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.517908 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.520696 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.520701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.520706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.520743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.524152 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.524942 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.524948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.524951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.525097 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.525488 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.525494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.525497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.525814 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.526099 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.526379 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.526587 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:27:17.528153 3762 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55506
I20260926 07:27:17.530287 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3834
I20260926 07:27:17.546679 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.531994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.2.147.193:42221)
0926 07:27:17.532182 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.532187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.532620 (+ 433us) server_negotiation.cc:407] Connection header received
0926 07:27:17.532856 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.532862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.532945 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.533116 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.534484 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.535553 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.537237 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.537840 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.537961 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.540663 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.540670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.540675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.540710 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.544064 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.544750 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.544755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.544759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.544894 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.545256 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.545261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.545264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.545553 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.545853 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.546112 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.546331 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:27:17.547365 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55262
W20260926 07:27:17.547905 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3834: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:17.548097 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3834
I20260926 07:27:17.577111 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3974
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:17.596451 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.578155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.2.147.194:32803)
0926 07:27:17.578325 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.578331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.578350 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:27:17.578412 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.578420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.578499 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.578622 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.580102 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.581275 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.583016 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.583793 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.583931 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.589133 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.589147 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.589164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.589204 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.593214 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.593958 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.593973 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.593977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.594161 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.594557 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.594563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.594566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.594990 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.595498 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.595826 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.596040 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:27:17.597204 4104 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60026
W20260926 07:27:17.597895 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3974: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:17.598078 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3974
I20260926 07:27:17.618229 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4113
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[3718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:17.635067 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:17.619039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.2.147.195:41145)
0926 07:27:17.619263 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:17.619270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:17.619316 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:27:17.619567 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:17.619574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:17.619660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:17.619763 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:17.620910 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.622008 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.623282 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:17.623871 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:17.623994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:17.629036 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:17.629052 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:17.629067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:17.629108 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:17.632412 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.633221 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.633228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.633230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.633375 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:17.633734 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:17.633737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:17.633739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:17.634069 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:17.634300 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:17.634615 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:27:17.634756 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:27:17.635767 4243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49490
W20260926 07:27:17.636425 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4113: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:17.636576 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 4113
I20260926 07:27:17.654500 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 3731
I20260926 07:27:17.655647 3803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:17.768965 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 3731
2026-09-26T07:27:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4353 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[4340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[4340](info): set up 2 sockets
Sep 26 07:27:17 dist-test-slave-1dwd krb5kdc[4340](info): commencing operation
krb5kdc: starting...
W20260926 07:27:18.831617 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:18 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:19.006532 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45347
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:19.321285 4353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:19.321610 4353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:19.321774 4353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:19.328842 4353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:19.328976 4353 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:27:19.329070 4353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:19.329205 4353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:19.329284 4353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:19.329370 4353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:19.338570 4353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:45347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:19.339787 4353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:19.341003 4353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:19.352047 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:19.352077 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:19.352095 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:19.353195 4353 server_base.cc:1034] running on GCE node
I20260926 07:27:19.353912 4353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:19.355238 4353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:19.356454 4353 hybrid_clock.cc:648] HybridClock initialized: now 1790407639356401 us; error 34 us; skew 500 ppm
I20260926 07:27:19.356771 4353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:19 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:19.359664 4353 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:19.360896 4353 webserver.cc:460] Webserver started at http://127.2.147.254:40133/ using document root <none> and password file <none>
I20260926 07:27:19.361531 4353 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:19.361619 4353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:19.361894 4353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:19.364872 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "61fa46e2424e420eaac4d1b8caed308b"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "96ecb6bc5f6d33459273668274611670"
server_key_iv: "ca0327c1846f271f48af948331f88865"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.365554 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "61fa46e2424e420eaac4d1b8caed308b"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "96ecb6bc5f6d33459273668274611670"
server_key_iv: "ca0327c1846f271f48af948331f88865"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.370398 4353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 07:27:19.373557 4368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:19.374795 4353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:19.374969 4353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "61fa46e2424e420eaac4d1b8caed308b"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "96ecb6bc5f6d33459273668274611670"
server_key_iv: "ca0327c1846f271f48af948331f88865"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.375152 4353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:19.385649 4353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:19.389181 4353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:19.389434 4353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:19.408906 4353 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:45347
I20260926 07:27:19.408924 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:45347 every 8 connection(s)
I20260926 07:27:19.410192 4353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:19.412273 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 4353
I20260926 07:27:19.412474 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:19.413152 2639 external_mini_cluster.cc:1428] Setting key bcc69c967547196fb8594ca85e4b3c5a
I20260926 07:27:19.414194 4430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:27:19 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:19.427862 4430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Bootstrap starting.
I20260926 07:27:19.430872 4430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:19.431890 4430 log.cc:824] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:19.434630 4430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: No bootstrap required, opened a new log
I20260926 07:27:19.434775 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:19.415014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.2.147.254:45347)
0926 07:27:19.415739 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:19.415749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:19.415788 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:19.417246 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:19.417266 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:19.417671 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:19.417980 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:19.419742 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.421001 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.422637 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.423339 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.423461 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:19.427955 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:19.427966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:19.427987 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:19.428034 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:19.430907 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.431629 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.431635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.431645 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.431798 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.432200 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.432204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.432206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.432694 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:19.433000 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:19.433452 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 07:27:19.433837 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":339,"threads_started":1}
I20260926 07:27:19.442788 4430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } }
I20260926 07:27:19.443145 4430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:19.443266 4430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fa46e2424e420eaac4d1b8caed308b, State: Initialized, Role: FOLLOWER
I20260926 07:27:19.443780 4430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } }
I20260926 07:27:19.444007 4430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:19.444144 4430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:19.444315 4430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:19.446328 4430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } }
I20260926 07:27:19.446771 4430 leader_election.cc:302] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61fa46e2424e420eaac4d1b8caed308b; no voters:
I20260926 07:27:19.447340 4430 leader_election.cc:288] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:19.447451 4435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:19.449158 4430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:19.451495 4435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [term 1 LEADER]: Becoming Leader. State: Replica: 61fa46e2424e420eaac4d1b8caed308b, State: Running, Role: LEADER
I20260926 07:27:19.452034 4435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } }
I20260926 07:27:19.454471 4437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61fa46e2424e420eaac4d1b8caed308b. Latest consensus state: current_term: 1 leader_uuid: "61fa46e2424e420eaac4d1b8caed308b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } } }
I20260926 07:27:19.454891 4437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:19.455256 4436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61fa46e2424e420eaac4d1b8caed308b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fa46e2424e420eaac4d1b8caed308b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45347 } } }
I20260926 07:27:19.455482 4436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b [sys.catalog]: This master's current role is: LEADER
W20260926 07:27:19.460045 4450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:19.460175 4450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:19.460301 4449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:19.461736 4449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:19.469355 4449 catalog_manager.cc:1347] Generated new cluster ID: a8b3c4b6dbb54d80a8ed174032683369
I20260926 07:27:19.469460 4449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:19.481755 4449 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:19.483507 4449 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:19.489753 4449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Generated new TSK 0
I20260926 07:27:19.490474 4449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:19.542047 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45347
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:19.851068 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:19.851398 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:19.851586 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:19.858479 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:19.858574 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:19.858678 4456 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:27:19.858757 4456 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:19.858876 4456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:19.868511 4456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:19.869794 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:19.871058 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:19.883790 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:19.883826 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:19.883860 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:19.884606 4456 server_base.cc:1034] running on GCE node
I20260926 07:27:19.885195 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:19.886068 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:19.887295 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790407639887250 us; error 54 us; skew 500 ppm
I20260926 07:27:19.887593 4456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:19 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:19.890517 4456 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:19.892006 4456 webserver.cc:460] Webserver started at http://127.2.147.193:46103/ using document root <none> and password file <none>
I20260926 07:27:19.892758 4456 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:19.892895 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:19.893211 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:19.896035 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "04a5b89da6e641c494f072a972e23df9"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "02cad91d17a49ff25d1d765e561e0887"
server_key_iv: "57d6d13139663adc557b3c058794435a"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.896754 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "04a5b89da6e641c494f072a972e23df9"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "02cad91d17a49ff25d1d765e561e0887"
server_key_iv: "57d6d13139663adc557b3c058794435a"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.901530 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 07:27:19.904691 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:19.906023 4456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:19.906217 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "04a5b89da6e641c494f072a972e23df9"
format_stamp: "Formatted at 2026-09-26 07:27:19 on dist-test-slave-1dwd"
server_key: "02cad91d17a49ff25d1d765e561e0887"
server_key_iv: "57d6d13139663adc557b3c058794435a"
server_key_version: "encryptionkey@0"
I20260926 07:27:19.906415 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:19.918843 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:19.922785 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:19.923076 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:19.925665 4456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:19.925796 4456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:19.925951 4456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:19.926066 4456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:27:19 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:19.948387 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:19.926113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:56227 (local address 127.2.147.254:45347)
0926 07:27:19.926277 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:19.926298 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:19.927674 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 07:27:19.929660 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:19.929665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:19.929757 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:19.929883 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:19.932067 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.933444 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.936417 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.936859 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.937004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:19.941673 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:19.941678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:19.941681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:19.941712 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:19.944755 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.945767 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.945775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.945781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.945960 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.946513 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.946518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.946522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.946814 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:19.946970 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:19.947763 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 07:27:19.948038 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:27:19.950127 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:19.926664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.193:56227)
0926 07:27:19.927495 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:19.927533 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:19.929385 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:19.930110 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:19.930118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:19.930901 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:19.931843 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:19.931878 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.933655 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.933662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:19.936243 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:19.936259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.937127 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.937136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:19.937346 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:19.938174 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:19.938210 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:19.941422 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:19.944898 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:19.944914 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:19.944938 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:19.945541 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:19.946121 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:19.946127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:19.946136 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:19.946366 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:19.946950 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:19.946958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:19.947382 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 07:27:19.947672 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 07:27:19.948251 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":341,"threads_started":1}
I20260926 07:27:19.971939 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:19.956061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:52451 (local address 127.2.147.254:45347)
0926 07:27:19.956259 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:19.956268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:19.957273 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 07:27:19.957436 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:19.957443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:19.957541 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:19.957673 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:19.958503 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.959822 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.962509 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.963263 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.963389 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:19.966212 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:19.966218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:19.966223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:19.966269 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:19.969365 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.970012 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.970019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.970023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.970169 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:19.970525 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:19.970530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:19.970533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:19.970787 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:19.970935 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:19.971343 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 07:27:19.971546 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:27:19.972028 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:19.956609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.193:52451)
0926 07:27:19.957188 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:19.957204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:19.957322 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:19.957684 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:19.957687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:19.958114 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:19.958373 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:19.958381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.960011 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.960023 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:19.962310 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:19.962326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:19.963705 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:19.963714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:19.963954 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:19.964780 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:19.964800 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:19.966014 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:19.969495 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:19.969500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:19.969504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:19.969878 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:19.970284 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:19.970289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:19.970293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:19.970410 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:19.970908 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:19.970913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:19.971088 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 07:27:19.971248 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:27:19.971578 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":325,"threads_started":1}
I20260926 07:27:19.980418 4456 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38107
I20260926 07:27:19.980451 4590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38107 every 8 connection(s)
I20260926 07:27:19.981607 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:19.983385 4591 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45347
I20260926 07:27:19.983614 4591 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:19.984230 4591 heartbeater.cc:511] Master 127.2.147.254:45347 requested a full tablet report, sending...
I20260926 07:27:19.985968 4385 ts_manager.cc:194] Registered new tserver with Master: 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107)
I20260926 07:27:19.987371 4385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:56227
I20260926 07:27:19.989599 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 4456
I20260926 07:27:19.989792 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:19.990388 2639 external_mini_cluster.cc:1428] Setting key 28e0f3373d8eb5d877375c747c3422ad
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:20.027594 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45347
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:20.340194 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:20.340456 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:20.340556 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:20.347681 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:20.347781 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:20.347872 4595 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:27:20.347923 4595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:20.347992 4595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:20.358472 4595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:20.359771 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:20.361019 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:20.373845 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:20.373871 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:20.373813 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:20.374574 4595 server_base.cc:1034] running on GCE node
I20260926 07:27:20.375146 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:20.376094 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:20.377314 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790407640377256 us; error 46 us; skew 500 ppm
I20260926 07:27:20.377610 4595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:20 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:20.390117 4595 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:20.391640 4595 webserver.cc:460] Webserver started at http://127.2.147.194:34753/ using document root <none> and password file <none>
I20260926 07:27:20.392596 4595 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:20.392789 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:20.393169 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:20.396124 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "63d5dd7d367644839d6ea25687ca2308"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "776c6de1a3577ab692252d54da88c54f"
server_key_iv: "a59369903319e0e36c82603ed576ffb8"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.396870 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "63d5dd7d367644839d6ea25687ca2308"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "776c6de1a3577ab692252d54da88c54f"
server_key_iv: "a59369903319e0e36c82603ed576ffb8"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.401903 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:27:20.405282 4610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:20.406518 4595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:20.406708 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "63d5dd7d367644839d6ea25687ca2308"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "776c6de1a3577ab692252d54da88c54f"
server_key_iv: "a59369903319e0e36c82603ed576ffb8"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.406903 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:20.442925 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:20.446640 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:20.446914 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:20.449437 4595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:20.449573 4595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:20.449743 4595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:20.449841 4595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:27:20 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:20.472090 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.450060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:59019 (local address 127.2.147.254:45347)
0926 07:27:20.450237 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:20.450243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:20.451694 (+ 1451us) server_negotiation.cc:407] Connection header received
0926 07:27:20.453783 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:20.453788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:20.453871 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:20.453991 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:20.456140 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.457256 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.459787 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.460269 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.460366 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.465676 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:20.465684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:20.465689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:20.465732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:20.468963 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.469895 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.469900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.469904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.470047 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.470504 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.470510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.470518 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.470770 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:20.470918 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:20.471709 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 07:27:20.471872 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:27:20.474648 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.450611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.194:59019)
0926 07:27:20.451499 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:20.451554 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:20.453477 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:20.454213 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:20.454224 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:20.454937 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:20.455893 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.455922 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.457407 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.457416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.459554 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.459566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.460518 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.460533 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.460895 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.461735 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:20.461775 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:20.465337 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:20.469092 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.469107 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.469122 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.469760 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.470170 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.470175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.470178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.470384 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.470892 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:20.470899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:20.471295 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 07:27:20.471622 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 07:27:20.472919 (+ 1297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":398,"threads_started":1}
I20260926 07:27:20.496167 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.482153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:34189 (local address 127.2.147.254:45347)
0926 07:27:20.482406 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:20.482414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:20.483014 (+ 600us) server_negotiation.cc:407] Connection header received
0926 07:27:20.483129 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:20.483138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:20.483274 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:20.483402 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:20.485102 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.486030 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.487813 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.488390 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.488522 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.490431 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:20.490437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:20.490444 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:20.490477 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:20.493589 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.494214 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.494221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.494224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.494353 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.494706 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.494713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.494716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.494998 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:20.495163 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:20.495593 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 07:27:20.495806 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:27:20.496403 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.482251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.194:34189)
0926 07:27:20.482899 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:20.482917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:20.483033 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:20.483597 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:20.483605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:20.484366 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:20.484912 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.484935 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.486205 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.486216 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.487669 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.487680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.488635 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.488645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.488781 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.489395 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:20.489411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:20.490261 (+ 850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:20.493725 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.493731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.493735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.494097 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.494459 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.494465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.494469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.494595 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.495114 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:20.495119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:20.495333 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:27:20.495506 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:27:20.495854 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":355,"threads_started":1}
I20260926 07:27:20.500226 4595 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:42909
I20260926 07:27:20.500254 4729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:42909 every 8 connection(s)
I20260926 07:27:20.501412 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:20.503196 4730 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45347
I20260926 07:27:20.503455 4730 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:20.504052 4730 heartbeater.cc:511] Master 127.2.147.254:45347 requested a full tablet report, sending...
I20260926 07:27:20.504756 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 4595
I20260926 07:27:20.504915 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:20.505439 2639 external_mini_cluster.cc:1428] Setting key 5d4647cb897d509cb80f077ef0a2ef65
I20260926 07:27:20.505596 4385 ts_manager.cc:194] Registered new tserver with Master: 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:20.506559 4385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:59019
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:20.543767 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45347
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:20.857769 4734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:20.858042 4734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:20.858140 4734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:20.865362 4734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:20.865476 4734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:20.865561 4734 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:27:20.865613 4734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:27:20.865674 4734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:20.875223 4734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:20.876477 4734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:20.877856 4734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:20.891397 4742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:20.891485 4739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:20.891433 4740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:20.891901 4734 server_base.cc:1034] running on GCE node
I20260926 07:27:20.892601 4734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:20.893481 4734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:20.894676 4734 hybrid_clock.cc:648] HybridClock initialized: now 1790407640894605 us; error 58 us; skew 500 ppm
I20260926 07:27:20.894990 4734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:20 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:20.898021 4734 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:20.899336 4734 webserver.cc:460] Webserver started at http://127.2.147.195:38339/ using document root <none> and password file <none>
I20260926 07:27:20.899923 4734 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:20.900013 4734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:20.900249 4734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:20.902917 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "ab43c8e4ff52817ee1c81ad43d7549fe"
server_key_iv: "f17c0dab4db6a446395a07aa1bf9a0c0"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.903621 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "ab43c8e4ff52817ee1c81ad43d7549fe"
server_key_iv: "f17c0dab4db6a446395a07aa1bf9a0c0"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.908419 4734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 07:27:20.911790 4749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:20.913058 4734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:20.913245 4734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e"
format_stamp: "Formatted at 2026-09-26 07:27:20 on dist-test-slave-1dwd"
server_key: "ab43c8e4ff52817ee1c81ad43d7549fe"
server_key_iv: "f17c0dab4db6a446395a07aa1bf9a0c0"
server_key_version: "encryptionkey@0"
I20260926 07:27:20.913440 4734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:20.937071 4734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:20.941452 4734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:20.941772 4734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:20.944408 4734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:20.944556 4734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:20.944694 4734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:20.944861 4734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 07:27:20 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:20.970043 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.945469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.195:53353)
0926 07:27:20.946319 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:20.946380 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:20.948309 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:20.949123 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:20.949134 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:20.949741 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:20.950686 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.950707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.952099 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.952110 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.954483 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.954494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.955710 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.955715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.955896 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.956763 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:20.956804 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:20.960703 (+ 3899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:20.965317 (+ 4614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.965331 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.965359 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.966190 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.966647 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.966652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.966657 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.966982 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.967571 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:20.967579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:20.968158 (+ 579us) client_negotiation.cc:769] Sending connection context
0926 07:27:20.968482 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 07:27:20.968996 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":379,"threads_started":1}
I20260926 07:27:20.971036 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.944903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:53353 (local address 127.2.147.254:45347)
0926 07:27:20.945099 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:20.945104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:20.946522 (+ 1418us) server_negotiation.cc:407] Connection header received
0926 07:27:20.948589 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:20.948596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:20.948691 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:20.948835 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:20.950861 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.951935 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.954641 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.955266 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.955418 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.961035 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:20.961044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:20.961051 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:20.961116 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:20.965128 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.966342 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.966348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.966352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.966511 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.967097 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.967101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.967105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.967429 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:20.967589 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:20.970657 (+ 3068us) server_negotiation.cc:299] Negotiation successful
0926 07:27:20.970823 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:27:20.988291 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.976191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:52347 (local address 127.2.147.254:45347)
0926 07:27:20.976369 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:20.976377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:20.977058 (+ 681us) server_negotiation.cc:407] Connection header received
0926 07:27:20.977259 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:20.977277 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:20.977342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:20.977413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:20.978792 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.979738 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.981054 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.981466 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.981600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.983304 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:20.983309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:20.983314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:20.983351 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:20.985783 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.986443 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.986450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.986457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.986579 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:20.986911 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:20.986917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:20.986920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:20.987151 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:20.987371 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:20.987751 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 07:27:20.987942 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:27:20.988363 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:20.976191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.2.147.195:52347)
0926 07:27:20.976899 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:20.976927 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:20.977129 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:20.977474 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:20.977481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:20.978200 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:20.978652 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.978667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.979938 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.979943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.980909 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:20.980919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:20.981639 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:20.981644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:20.981794 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:20.982382 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:20.982397 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:20.983116 (+ 719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:20.985919 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.985925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.985928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.986314 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.986688 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:20.986692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:20.986696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:20.986811 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:20.987342 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:20.987346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:20.987513 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 07:27:20.987645 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:27:20.987936 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":334,"threads_started":1}
I20260926 07:27:20.989905 4591 heartbeater.cc:503] Master 127.2.147.254:45347 was elected leader, sending a full tablet report...
I20260926 07:27:21.007239 4734 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:35405
I20260926 07:27:21.007292 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:35405 every 8 connection(s)
I20260926 07:27:21.008477 4734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:21.010624 4869 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45347
I20260926 07:27:21.010815 4869 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:21.011577 4869 heartbeater.cc:511] Master 127.2.147.254:45347 requested a full tablet report, sending...
I20260926 07:27:21.012923 4384 ts_manager.cc:194] Registered new tserver with Master: db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405)
I20260926 07:27:21.013803 4384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:53353
I20260926 07:27:21.018030 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 4734
I20260926 07:27:21.018164 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:21.018549 2639 external_mini_cluster.cc:1428] Setting key 8169e2ced578ab54cbe230fe175f63d4
I20260926 07:27:21.021265 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:27:21 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:27:21 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:21.065438 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.052138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.2.147.254:45347)
0926 07:27:21.052308 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.052312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.052668 (+ 356us) server_negotiation.cc:407] Connection header received
0926 07:27:21.052856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.052862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.052956 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.053067 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:21.054349 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.055082 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.056293 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.056687 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.056777 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.060342 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:21.060346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:21.060349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:21.060380 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:21.062874 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:21.063572 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:21.063584 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:21.063587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:21.063722 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:21.064094 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:21.064098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:21.064100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:21.064358 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:21.064590 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.064893 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.065122 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:21.069235 4384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:21.076295 4384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:21.106128 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.090595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:42909 (local address 127.0.0.1:34324)
0926 07:27:21.091786 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.091832 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.092000 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.096086 (+ 4086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.096097 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.096132 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.096676 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.096692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.099399 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.099408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.102451 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.102473 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.104881 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.104889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.105178 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.105254 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.105381 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.105642 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":505,"threads_started":1}
I20260926 07:27:21.107700 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.090710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.2.147.194:42909)
0926 07:27:21.091600 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.091610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.095304 (+ 3694us) server_negotiation.cc:407] Connection header received
0926 07:27:21.095439 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.095452 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.095803 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.096920 (+ 1117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.097167 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.099154 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.102727 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.104618 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.106777 (+ 2159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.107026 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.107123 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.107379 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":634,"threads_started":1}
I20260926 07:27:21.114702 4660 tablet_service.cc:1467] Processing CreateTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 (DEFAULT_TABLE table=test-table [id=003a3cab12654cbeb6c62feab0704e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.116427 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b17c34bc5034fc9a9f11057b657d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.117208 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.091089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35405 (local address 127.0.0.1:59670)
0926 07:27:21.092592 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.092625 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.092749 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.109352 (+ 16603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.109359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.109399 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.109932 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.109945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.112214 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.112221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.114366 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.114380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.116375 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.116382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.116563 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.116599 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.116676 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.116852 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":18413,"threads_started":1}
I20260926 07:27:21.118351 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.091845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.2.147.195:35405)
0926 07:27:21.108514 (+ 16669us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.108527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.108564 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:27:21.108744 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.108757 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.109090 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.109324 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.110156 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.112012 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.114571 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.116119 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.117520 (+ 1401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.117773 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.117845 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.118009 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16442,"thread_start_us":16201,"threads_started":1}
I20260926 07:27:21.122036 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 (DEFAULT_TABLE table=test-table [id=003a3cab12654cbeb6c62feab0704e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.123278 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b17c34bc5034fc9a9f11057b657d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.128744 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.091163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38107 (local address 127.0.0.1:37186)
0926 07:27:21.106491 (+ 15328us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.106508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.106616 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.110575 (+ 3959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.110579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.110602 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.111172 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.111185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.124269 (+ 13084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.124275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.126080 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.126099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.127953 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.127959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.128159 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.128196 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.128269 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.128437 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15237,"thread_start_us":1159,"threads_started":1}
I20260926 07:27:21.129109 4888 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Bootstrap starting.
I20260926 07:27:21.129966 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.091796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.2.147.193:38107)
0926 07:27:21.109972 (+ 18176us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.109980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.110006 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:27:21.110102 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.110109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.110338 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.110555 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.122617 (+ 12062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.124113 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.126258 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.127765 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.129306 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.129456 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.129509 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.129625 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18020,"thread_start_us":18962,"threads_started":1}
I20260926 07:27:21.132632 4888 tablet_bootstrap.cc:654] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.132867 4521 tablet_service.cc:1467] Processing CreateTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 (DEFAULT_TABLE table=test-table [id=003a3cab12654cbeb6c62feab0704e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.133930 4888 log.cc:824] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:21.134043 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b17c34bc5034fc9a9f11057b657d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.137269 4890 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Bootstrap starting.
I20260926 07:27:21.137423 4888 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: No bootstrap required, opened a new log
I20260926 07:27:21.137758 4888 ts_tablet_manager.cc:1397] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 07:27:21.140662 4890 tablet_bootstrap.cc:654] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.141978 4890 log.cc:824] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:21.144757 4890 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: No bootstrap required, opened a new log
I20260926 07:27:21.145001 4890 ts_tablet_manager.cc:1397] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:27:21.145479 4888 raft_consensus.cc:348] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.145814 4888 raft_consensus.cc:374] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.145938 4888 raft_consensus.cc:729] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.146930 4888 consensus_queue.cc:260] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.148012 4730 heartbeater.cc:503] Master 127.2.147.254:45347 was elected leader, sending a full tablet report...
I20260926 07:27:21.148562 4888 ts_tablet_manager.cc:1428] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Time spent starting tablet: real 0.011s user 0.000s sys 0.009s
I20260926 07:27:21.149224 4892 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Bootstrap starting.
I20260926 07:27:21.152875 4892 tablet_bootstrap.cc:654] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.154093 4892 log.cc:824] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:21.157590 4892 tablet_bootstrap.cc:492] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: No bootstrap required, opened a new log
I20260926 07:27:21.157861 4892 ts_tablet_manager.cc:1397] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:27:21.158215 4890 raft_consensus.cc:348] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.158629 4890 raft_consensus.cc:374] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.158726 4890 raft_consensus.cc:729] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.159360 4890 consensus_queue.cc:260] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.160336 4869 heartbeater.cc:503] Master 127.2.147.254:45347 was elected leader, sending a full tablet report...
I20260926 07:27:21.160936 4890 ts_tablet_manager.cc:1428] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260926 07:27:21.166021 4892 raft_consensus.cc:348] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.166330 4892 raft_consensus.cc:374] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.166404 4892 raft_consensus.cc:729] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5b89da6e641c494f072a972e23df9, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.166908 4892 consensus_queue.cc:260] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.168669 4892 ts_tablet_manager.cc:1428] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260926 07:27:21.234038 4592 tablet.cc:2369] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:21.238014 4893 raft_consensus.cc:482] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:21.238302 4893 raft_consensus.cc:504] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.239986 4893 leader_election.cc:288] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405), 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107)
I20260926 07:27:21.248696 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.240906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:51689 (local address 127.2.147.193:38107)
0926 07:27:21.241113 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.241118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.241493 (+ 375us) server_negotiation.cc:407] Connection header received
0926 07:27:21.241586 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.241590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.241660 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.241744 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.242373 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.243369 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.246187 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.247437 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.247587 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.247660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.248268 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.248398 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:27:21.248903 4898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.240409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38107 (local address 127.2.147.194:51689)
0926 07:27:21.241404 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.241416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.241494 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.241779 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.241782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.241819 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.242240 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.242248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.243493 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.243499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.246024 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.246038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.247707 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.247718 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.248009 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.248122 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.248227 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.248365 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":197,"spinlock_wait_cycles":1920,"thread_start_us":458,"threads_started":1}
I20260926 07:27:21.249258 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.240174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35405 (local address 127.2.147.194:60711)
0926 07:27:21.240989 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.241022 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.241239 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.241608 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.241617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.241634 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.242034 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.242047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.243363 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.243366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.246025 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.246038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.247726 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.247730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.248018 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.248123 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.248268 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.248367 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":284,"thread_start_us":367,"threads_started":1}
I20260926 07:27:21.249488 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.240386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:60711 (local address 127.2.147.195:35405)
0926 07:27:21.240589 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.240611 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.241119 (+ 508us) server_negotiation.cc:407] Connection header received
0926 07:27:21.241395 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.241399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.241482 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.241578 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.242184 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.243258 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.246183 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.247427 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.247587 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.247660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.248304 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.248459 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:27:21.250134 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b17c34bc5034fc9a9f11057b657d021" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9" is_pre_election: true
I20260926 07:27:21.250470 4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b17c34bc5034fc9a9f11057b657d021" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" is_pre_election: true
I20260926 07:27:21.250716 4546 raft_consensus.cc:2454] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 0.
I20260926 07:27:21.250983 4819 raft_consensus.cc:2454] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 0.
I20260926 07:27:21.251533 4611 leader_election.cc:302] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5b89da6e641c494f072a972e23df9, 63d5dd7d367644839d6ea25687ca2308; no voters:
I20260926 07:27:21.252214 4893 raft_consensus.cc:2781] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:21.252379 4893 raft_consensus.cc:482] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:21.252475 4893 raft_consensus.cc:3037] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:27:21.253239 4731 tablet.cc:2369] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:21.255251 4893 raft_consensus.cc:504] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.255934 4893 leader_election.cc:288] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 election: Requested vote from peers db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405), 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107)
I20260926 07:27:21.256580 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b17c34bc5034fc9a9f11057b657d021" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9"
I20260926 07:27:21.256798 4546 raft_consensus.cc:3037] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.258658 4819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b17c34bc5034fc9a9f11057b657d021" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e"
I20260926 07:27:21.258869 4819 raft_consensus.cc:3037] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.260072 4546 raft_consensus.cc:2454] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 1.
W20260926 07:27:21.260138 4870 tablet.cc:2369] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:21.260540 4611 leader_election.cc:302] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5b89da6e641c494f072a972e23df9, 63d5dd7d367644839d6ea25687ca2308; no voters:
I20260926 07:27:21.260854 4893 raft_consensus.cc:2781] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:21.262737 4819 raft_consensus.cc:2454] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 1.
I20260926 07:27:21.263458 4893 raft_consensus.cc:686] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 1 LEADER]: Becoming Leader. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Running, Role: LEADER
I20260926 07:27:21.264025 4893 consensus_queue.cc:237] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } }
I20260926 07:27:21.269686 4385 catalog_manager.cc:5495] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "63d5dd7d367644839d6ea25687ca2308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:27:21 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:21.342973 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.330005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45347 (local address 127.0.0.1:58182)
0926 07:27:21.330659 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.330674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.330800 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.331085 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.331088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.331805 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:21.332049 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.332057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.333036 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.333039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.334024 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.334036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.334714 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.334718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.334831 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.335435 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:21.335480 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:21.337680 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:21.340521 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:21.340531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:21.340539 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:21.340919 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:21.341264 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:21.341270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:21.341273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:21.341419 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:21.341906 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:21.341914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:21.342076 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.342240 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.342522 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":408,"threads_started":1}
I20260926 07:27:21.342988 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.329918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.2.147.254:45347)
0926 07:27:21.330117 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.330124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.330763 (+ 639us) server_negotiation.cc:407] Connection header received
0926 07:27:21.330898 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.330902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.330959 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.331042 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:21.332175 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.332907 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.334143 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.334533 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.334619 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.337850 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:21.337855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:21.337858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:21.337882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:21.340379 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:21.341027 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:21.341035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:21.341039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:21.341162 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:21.341524 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:21.341531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:21.341534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:21.341781 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:21.341925 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.342318 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.342639 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":74}
I20260926 07:27:21.348003 4385 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:58182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:27:21.349068 4385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:21.355801 4521 tablet_service.cc:1467] Processing CreateTablet for tablet 614c0b12032c43afa5956ea7eeca3edb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e84758b16c2442bb83d0d86d9b2d6e06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:21.355816 4660 tablet_service.cc:1467] Processing CreateTablet for tablet 614c0b12032c43afa5956ea7eeca3edb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e84758b16c2442bb83d0d86d9b2d6e06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:21.356135 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 614c0b12032c43afa5956ea7eeca3edb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e84758b16c2442bb83d0d86d9b2d6e06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:27:21.356649 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c0b12032c43afa5956ea7eeca3edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.356706 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c0b12032c43afa5956ea7eeca3edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.356874 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c0b12032c43afa5956ea7eeca3edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.362051 4892 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: Bootstrap starting.
I20260926 07:27:21.363813 4892 tablet_bootstrap.cc:654] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.363888 4890 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: Bootstrap starting.
I20260926 07:27:21.365163 4888 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: Bootstrap starting.
I20260926 07:27:21.365556 4890 tablet_bootstrap.cc:654] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.366043 4892 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: No bootstrap required, opened a new log
I20260926 07:27:21.366140 4892 ts_tablet_manager.cc:1397] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:27:21.366667 4892 raft_consensus.cc:348] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.366840 4892 raft_consensus.cc:374] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.366897 4892 raft_consensus.cc:729] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5b89da6e641c494f072a972e23df9, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.367028 4888 tablet_bootstrap.cc:654] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.367103 4892 consensus_queue.cc:260] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.367736 4892 ts_tablet_manager.cc:1428] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:21.367592 4896 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.367978 4896 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.369462 4888 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: No bootstrap required, opened a new log
I20260926 07:27:21.369619 4888 ts_tablet_manager.cc:1397] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 07:27:21.369804 4890 tablet_bootstrap.cc:492] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: No bootstrap required, opened a new log
I20260926 07:27:21.369948 4890 ts_tablet_manager.cc:1397] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 07:27:21.370410 4888 raft_consensus.cc:348] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.370645 4888 raft_consensus.cc:374] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.370579 4890 raft_consensus.cc:348] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.370760 4888 raft_consensus.cc:729] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.370836 4890 raft_consensus.cc:374] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.370936 4890 raft_consensus.cc:729] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.370994 4888 consensus_queue.cc:260] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.371155 4890 consensus_queue.cc:260] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.371414 4893 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.371709 4893 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.371835 4890 ts_tablet_manager.cc:1428] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 07:27:21.371665 4895 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.372037 4895 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.371584 4888 ts_tablet_manager.cc:1428] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:27:21.394428 4895 raft_consensus.cc:482] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:21.394708 4895 raft_consensus.cc:504] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.396450 4895 leader_election.cc:288] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:21.402022 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.396739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:42909 (local address 127.2.147.195:44131)
0926 07:27:21.397017 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.397037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.397172 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.397523 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.397527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.397553 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.397761 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.397769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.398755 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.398763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.400249 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.400260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.401309 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.401316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.401497 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.401520 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.401621 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.401746 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 07:27:21.402017 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.396899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:44131 (local address 127.2.147.194:42909)
0926 07:27:21.397078 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.397086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.397105 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:27:21.397322 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.397327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.397410 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.397503 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.397891 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.398630 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.400401 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.401164 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.401332 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.401407 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.401655 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.401770 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:27:21.402657 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.396746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:36129 (local address 127.2.147.193:38107)
0926 07:27:21.396922 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.396928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.397597 (+ 669us) server_negotiation.cc:407] Connection header received
0926 07:27:21.397798 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.397804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.397870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.397964 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:21.398432 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.399167 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.400895 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.401660 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.401852 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.401921 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.402248 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.402393 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:21.402918 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c0b12032c43afa5956ea7eeca3edb" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308" is_pre_election: true
I20260926 07:27:21.403311 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.396782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38107 (local address 127.2.147.195:36129)
0926 07:27:21.397497 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.397510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.397664 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.397977 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.397981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.398003 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:21.398293 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.398301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.399319 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.399327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.400744 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.400754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.401880 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.401886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.402038 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.402060 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.402831 (+ 771us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.402962 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
I20260926 07:27:21.403442 4680 raft_consensus.cc:2454] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:21.404112 4751 leader_election.cc:302] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d5dd7d367644839d6ea25687ca2308, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:21.404107 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c0b12032c43afa5956ea7eeca3edb" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9" is_pre_election: true
I20260926 07:27:21.404402 4546 raft_consensus.cc:2454] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:21.404517 4895 raft_consensus.cc:2781] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:21.404675 4895 raft_consensus.cc:482] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:21.404784 4895 raft_consensus.cc:3037] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.406827 4895 raft_consensus.cc:504] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.407430 4895 leader_election.cc:288] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Requested vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:21.407815 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c0b12032c43afa5956ea7eeca3edb" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9"
I20260926 07:27:21.407858 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c0b12032c43afa5956ea7eeca3edb" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308"
I20260926 07:27:21.408035 4546 raft_consensus.cc:3037] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.408035 4680 raft_consensus.cc:3037] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.411039 4680 raft_consensus.cc:2454] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:21.411003 4546 raft_consensus.cc:2454] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:21.411450 4751 leader_election.cc:302] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d5dd7d367644839d6ea25687ca2308, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:21.411718 4895 raft_consensus.cc:2781] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:21.412088 4895 raft_consensus.cc:686] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 LEADER]: Becoming Leader. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Running, Role: LEADER
I20260926 07:27:21.412698 4895 consensus_queue.cc:237] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.416500 4911 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: TxnStatusTablet state changed. Reason: New leader db8fb9bbdb014c63b017d6fb23bd1d7e. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.416880 4911 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:21.417501 4913 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:27:21.417831 4913 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:27:21.418207 4385 catalog_manager.cc:5495] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195). New cstate: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:21.441562 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.435563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35405 (local address 127.0.0.1:59682)
0926 07:27:21.435923 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.435939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.436049 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.436516 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.436519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.436528 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:21.436668 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.436674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.438132 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.438135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.439040 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.439050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.439987 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.439991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.440136 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.440195 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.441157 (+ 962us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.441175 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.441241 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.441353 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260926 07:27:21.441655 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.435558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.2.147.195:35405)
0926 07:27:21.435728 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.435734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.436015 (+ 281us) server_negotiation.cc:407] Connection header received
0926 07:27:21.436245 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.436252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.436359 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.436500 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:21.436795 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.438009 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.439170 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.439856 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.439981 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.440379 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.441024 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.441191 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.441266 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.441355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:27:21.448755 4680 raft_consensus.cc:1261] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:21.448799 4546 raft_consensus.cc:1261] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:21.449661 4911 consensus_queue.cc:1035] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:21.450006 4895 consensus_queue.cc:1035] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:21.463603 4893 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: TxnStatusTablet state changed. Reason: New leader db8fb9bbdb014c63b017d6fb23bd1d7e. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.463968 4893 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.466161 4916 mvcc.cc:205] Tried to move back new op lower bound from 7333509699365031936 to 7333509699230457856. Current Snapshot: MvccSnapshot[applied={T|T < 7333509699365031936}]
I20260926 07:27:21.466679 4896 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: TxnStatusTablet state changed. Reason: New leader db8fb9bbdb014c63b017d6fb23bd1d7e. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.466965 4896 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.471676 4918 mvcc.cc:205] Tried to move back new op lower bound from 7333509699365031936 to 7333509699230457856. Current Snapshot: MvccSnapshot[applied={T|T < 7333509699365031936}]
I20260926 07:27:21.472837 4917 mvcc.cc:205] Tried to move back new op lower bound from 7333509699365031936 to 7333509699230457856. Current Snapshot: MvccSnapshot[applied={T|T < 7333509699365031936}]
I20260926 07:27:21.477226 4896 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.477483 4896 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 04a5b89da6e641c494f072a972e23df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.478513 4895 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.478823 4895 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:21.479175 4893 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.479501 4893 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P 63d5dd7d367644839d6ea25687ca2308: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:27:21.481328 4911 tablet_replica.cc:439] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } } }
I20260926 07:27:21.481559 4911 tablet_replica.cc:442] T 614c0b12032c43afa5956ea7eeca3edb P db8fb9bbdb014c63b017d6fb23bd1d7e: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:27:21.492379 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.486345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.2.147.194:42909)
0926 07:27:21.486683 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.486690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.486711 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:27:21.486939 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.486943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.487013 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.487140 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:21.487554 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.488288 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.490196 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.490586 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.490678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.491278 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.491793 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.491900 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.492030 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.492096 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260926 07:27:21.509490 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.503379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:49511 (local address 127.2.147.195:35405)
0926 07:27:21.503607 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.503615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.504276 (+ 661us) server_negotiation.cc:407] Connection header received
0926 07:27:21.504558 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.504578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.504661 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.504723 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:21.505390 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.506095 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.507577 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.508002 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.508170 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.508639 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.508782 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.508876 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.509087 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.509186 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:21.509608 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.503521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35405 (local address 127.2.147.194:49511)
0926 07:27:21.504170 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.504194 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.504389 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.504919 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.504922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.504931 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:21.505241 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.505249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.506230 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.506235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.507446 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.507457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.508213 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.508222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.508350 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.508398 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.508877 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.508890 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.509058 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.509191 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260926 07:27:21.519749 4546 raft_consensus.cc:1261] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Refusing update from remote peer 63d5dd7d367644839d6ea25687ca2308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:21.519985 4819 raft_consensus.cc:1261] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Refusing update from remote peer 63d5dd7d367644839d6ea25687ca2308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:21.520555 4899 consensus_queue.cc:1035] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:21.521207 4893 consensus_queue.cc:1035] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:21.565351 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.558603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:58929 (local address 127.2.147.194:42909)
0926 07:27:21.559010 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:21.559016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:21.559444 (+ 428us) server_negotiation.cc:407] Connection header received
0926 07:27:21.559558 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:21.559570 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:21.559648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:21.559740 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:21.560150 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.561203 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.563438 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.563845 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.563968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.564507 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.564668 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.564764 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:21.564962 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:27:21.565050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260926 07:27:21.565805 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:21.558673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:42909 (local address 127.2.147.195:58929)
0926 07:27:21.559350 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:21.559365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:21.559466 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:21.559758 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:21.559761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:21.559770 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:21.560009 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.560016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.561363 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.561369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.563313 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:21.563327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:21.564052 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:21.564060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:21.564268 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:21.564339 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:21.564776 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:21.564789 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:27:21.564908 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:27:21.565393 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":340,"threads_started":1}
I20260926 07:27:21.602074 4384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:21.602396 4384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:21.606660 4384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Sending DeleteTablet for 3 replicas of tablet 4b17c34bc5034fc9a9f11057b657d021
I20260926 07:27:21.607925 4799 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260926 07:27:21.608676 4521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37186
I20260926 07:27:21.608892 4937 tablet_replica.cc:330] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: stopping tablet replica
I20260926 07:27:21.609345 4937 raft_consensus.cc:2229] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:21.609529 4660 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b17c34bc5034fc9a9f11057b657d021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260926 07:27:21.609768 4937 raft_consensus.cc:2258] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:21.612002 4937 ts_tablet_manager.cc:1905] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:21.612538 4384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:27:21.613178 4939 tablet_replica.cc:330] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: stopping tablet replica
W20260926 07:27:21.613232 4384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:21.613560 4939 raft_consensus.cc:2229] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:21.614177 4939 raft_consensus.cc:2258] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:21.620766 4938 tablet_replica.cc:330] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: stopping tablet replica
I20260926 07:27:21.621323 4938 raft_consensus.cc:2229] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:21.621744 4938 raft_consensus.cc:2258] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:21.622529 4521 tablet_service.cc:1467] Processing CreateTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (DEFAULT_TABLE table=test-table [id=3f1d6b67b7b4496384cb221cfa848150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.622789 4660 tablet_service.cc:1467] Processing CreateTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (DEFAULT_TABLE table=test-table [id=3f1d6b67b7b4496384cb221cfa848150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.623106 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c0f2b400b84d06a1a9ecaaaf503e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.623217 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c0f2b400b84d06a1a9ecaaaf503e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.625031 4939 ts_tablet_manager.cc:1905] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:21.628273 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (DEFAULT_TABLE table=test-table [id=3f1d6b67b7b4496384cb221cfa848150]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:21.632264 4888 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Bootstrap starting.
I20260926 07:27:21.632364 4939 ts_tablet_manager.cc:1918] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:21.632503 4939 log.cc:1196] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/4b17c34bc5034fc9a9f11057b657d021
I20260926 07:27:21.633095 4939 ts_tablet_manager.cc:1939] T 4b17c34bc5034fc9a9f11057b657d021 P 63d5dd7d367644839d6ea25687ca2308: Deleting consensus metadata
I20260926 07:27:21.633431 4938 ts_tablet_manager.cc:1905] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:21.634064 4888 tablet_bootstrap.cc:654] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.634922 4370 catalog_manager.cc:4841] TS 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909): tablet 4b17c34bc5034fc9a9f11057b657d021 (table test-table [id=003a3cab12654cbeb6c62feab0704e75]) successfully deleted
I20260926 07:27:21.638794 4892 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Bootstrap starting.
I20260926 07:27:21.640812 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c0f2b400b84d06a1a9ecaaaf503e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:21.640908 4892 tablet_bootstrap.cc:654] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.647658 4890 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Bootstrap starting.
I20260926 07:27:21.649420 4890 tablet_bootstrap.cc:654] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:21.658901 4938 ts_tablet_manager.cc:1918] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:21.659096 4938 log.cc:1196] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/4b17c34bc5034fc9a9f11057b657d021
I20260926 07:27:21.660575 4937 ts_tablet_manager.cc:1918] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:21.660753 4937 log.cc:1196] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/4b17c34bc5034fc9a9f11057b657d021
I20260926 07:27:21.661109 4888 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: No bootstrap required, opened a new log
I20260926 07:27:21.661360 4888 ts_tablet_manager.cc:1397] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.014s
I20260926 07:27:21.661589 4937 ts_tablet_manager.cc:1939] T 4b17c34bc5034fc9a9f11057b657d021 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting consensus metadata
I20260926 07:27:21.662568 4888 raft_consensus.cc:348] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.663004 4888 raft_consensus.cc:374] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.663151 4888 raft_consensus.cc:729] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.663504 4371 catalog_manager.cc:4841] TS db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405): tablet 4b17c34bc5034fc9a9f11057b657d021 (table test-table [id=003a3cab12654cbeb6c62feab0704e75]) successfully deleted
I20260926 07:27:21.663928 4892 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: No bootstrap required, opened a new log
I20260926 07:27:21.664732 4892 ts_tablet_manager.cc:1397] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20260926 07:27:21.665150 4888 consensus_queue.cc:260] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.665484 4892 raft_consensus.cc:348] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.665717 4892 raft_consensus.cc:374] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.665791 4892 raft_consensus.cc:729] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5b89da6e641c494f072a972e23df9, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.666005 4892 consensus_queue.cc:260] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.666558 4892 ts_tablet_manager.cc:1428] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:21.668103 4938 ts_tablet_manager.cc:1939] T 4b17c34bc5034fc9a9f11057b657d021 P 04a5b89da6e641c494f072a972e23df9: Deleting consensus metadata
I20260926 07:27:21.669895 4888 ts_tablet_manager.cc:1428] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260926 07:27:21.672118 4369 catalog_manager.cc:4841] TS 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107): tablet 4b17c34bc5034fc9a9f11057b657d021 (table test-table [id=003a3cab12654cbeb6c62feab0704e75]) successfully deleted
I20260926 07:27:21.675586 4890 tablet_bootstrap.cc:492] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: No bootstrap required, opened a new log
I20260926 07:27:21.675788 4890 ts_tablet_manager.cc:1397] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.009s
I20260926 07:27:21.676501 4890 raft_consensus.cc:348] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.676802 4890 raft_consensus.cc:374] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:21.676915 4890 raft_consensus.cc:729] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Initialized, Role: FOLLOWER
I20260926 07:27:21.677199 4890 consensus_queue.cc:260] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.677852 4890 ts_tablet_manager.cc:1428] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:21.957311 4895 raft_consensus.cc:482] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:21.957532 4895 raft_consensus.cc:504] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.958199 4895 leader_election.cc:288] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:21.958779 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c0f2b400b84d06a1a9ecaaaf503e97" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308" is_pre_election: true
I20260926 07:27:21.958801 4544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c0f2b400b84d06a1a9ecaaaf503e97" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9" is_pre_election: true
I20260926 07:27:21.959015 4680 raft_consensus.cc:2454] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:21.959067 4544 raft_consensus.cc:2454] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:21.959518 4751 leader_election.cc:302] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d5dd7d367644839d6ea25687ca2308, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:21.959820 4895 raft_consensus.cc:2781] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:21.959964 4895 raft_consensus.cc:482] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:21.960071 4895 raft_consensus.cc:3037] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.961956 4895 raft_consensus.cc:504] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.962515 4895 leader_election.cc:288] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Requested vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:21.963020 4544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c0f2b400b84d06a1a9ecaaaf503e97" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9"
I20260926 07:27:21.963056 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c0f2b400b84d06a1a9ecaaaf503e97" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308"
I20260926 07:27:21.963249 4544 raft_consensus.cc:3037] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.963291 4680 raft_consensus.cc:3037] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:21.965590 4680 raft_consensus.cc:2454] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:21.965973 4751 leader_election.cc:302] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d5dd7d367644839d6ea25687ca2308, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:21.966228 4895 raft_consensus.cc:2781] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:21.966378 4895 raft_consensus.cc:686] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 LEADER]: Becoming Leader. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Running, Role: LEADER
I20260926 07:27:21.966647 4544 raft_consensus.cc:2454] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:21.966626 4895 consensus_queue.cc:237] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:21.969415 4384 catalog_manager.cc:5495] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195). New cstate: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:22.029968 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:22.016221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.2.147.195:35405)
0926 07:27:22.019804 (+ 3583us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:22.019810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:22.023808 (+ 3998us) server_negotiation.cc:407] Connection header received
0926 07:27:22.024045 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:22.024051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:22.024129 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:22.024232 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:22.024720 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.025780 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.027672 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.028344 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.028450 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:22.028945 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:22.029089 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:22.029163 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:22.029592 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 07:27:22.029663 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3422,"thread_start_us":3334,"threads_started":1}
I20260926 07:27:22.047164 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:22.042586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:53611 (local address 127.2.147.195:35405)
0926 07:27:22.042758 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:22.042764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:22.042783 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:27:22.042930 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:22.042935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:22.043005 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:22.043085 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:22.043470 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.044398 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.045662 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.046051 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.046172 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:22.046502 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:22.046608 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:22.046699 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:22.046799 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:27:22.046859 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:27:22.047242 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:22.042525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35405 (local address 127.2.147.195:53611)
0926 07:27:22.042701 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:22.042725 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:22.042805 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:22.043104 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:22.043108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:22.043119 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:27:22.043320 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:22.043328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.044521 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.044526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:22.045518 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:22.045527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:22.046187 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:22.046192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:22.046330 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:22.046359 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:27:22.046705 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:27:22.046717 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:27:22.046785 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:27:22.046919 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":11}
I20260926 07:27:22.055722 4544 raft_consensus.cc:1261] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:22.055723 4680 raft_consensus.cc:1261] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:22.056322 4928 consensus_queue.cc:1035] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:22.056979 4928 consensus_queue.cc:1035] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:27:22.078002 4916 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:27:23.661844 4385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:23.662056 4385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:23.665982 4385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Sending DeleteTablet for 3 replicas of tablet 61c0f2b400b84d06a1a9ecaaaf503e97
I20260926 07:27:23.666735 4521 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37186
I20260926 07:27:23.666734 4660 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260926 07:27:23.666842 4799 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c0f2b400b84d06a1a9ecaaaf503e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260926 07:27:23.679765 4972 tablet_replica.cc:330] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: stopping tablet replica
I20260926 07:27:23.680219 4972 raft_consensus.cc:2229] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:23.680603 4972 raft_consensus.cc:2258] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.681252 4973 tablet_replica.cc:330] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: stopping tablet replica
I20260926 07:27:23.681427 4971 tablet_replica.cc:330] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: stopping tablet replica
I20260926 07:27:23.681650 4971 raft_consensus.cc:2229] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:23.681875 4971 raft_consensus.cc:2258] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.681892 4973 raft_consensus.cc:2229] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:23.682461 4973 raft_consensus.cc:2258] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.682647 4972 ts_tablet_manager.cc:1905] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:23.682776 4971 ts_tablet_manager.cc:1905] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:23.683359 4973 ts_tablet_manager.cc:1905] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:23.689282 4972 ts_tablet_manager.cc:1918] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:23.689476 4972 log.cc:1196] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/61c0f2b400b84d06a1a9ecaaaf503e97
I20260926 07:27:23.689832 4973 ts_tablet_manager.cc:1918] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:23.689967 4973 log.cc:1196] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/61c0f2b400b84d06a1a9ecaaaf503e97
I20260926 07:27:23.690101 4972 ts_tablet_manager.cc:1939] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 63d5dd7d367644839d6ea25687ca2308: Deleting consensus metadata
I20260926 07:27:23.690454 4973 ts_tablet_manager.cc:1939] T 61c0f2b400b84d06a1a9ecaaaf503e97 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting consensus metadata
I20260926 07:27:23.690478 4971 ts_tablet_manager.cc:1918] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:23.690999 4971 log.cc:1196] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/61c0f2b400b84d06a1a9ecaaaf503e97
I20260926 07:27:23.691462 4370 catalog_manager.cc:4841] TS 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909): tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (table test-table [id=3f1d6b67b7b4496384cb221cfa848150]) successfully deleted
I20260926 07:27:23.691561 4971 ts_tablet_manager.cc:1939] T 61c0f2b400b84d06a1a9ecaaaf503e97 P 04a5b89da6e641c494f072a972e23df9: Deleting consensus metadata
I20260926 07:27:23.691462 4371 catalog_manager.cc:4841] TS db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405): tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (table test-table [id=3f1d6b67b7b4496384cb221cfa848150]) successfully deleted
I20260926 07:27:23.692626 4369 catalog_manager.cc:4841] TS 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107): tablet 61c0f2b400b84d06a1a9ecaaaf503e97 (table test-table [id=3f1d6b67b7b4496384cb221cfa848150]) successfully deleted
Sep 26 07:27:23 dist-test-slave-1dwd krb5kdc[4340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:27:23 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:23.724324 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:23.710797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.2.147.254:45347)
0926 07:27:23.711414 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:23.711420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:23.711446 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:27:23.711665 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:23.711671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:23.711764 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:23.711876 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:23.713114 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:23.713874 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:23.715134 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:23.715609 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:23.715751 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:23.719512 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:23.719517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:23.719521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:23.719551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:23.721931 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:23.722504 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:23.722507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:23.722509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:23.722634 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:23.722980 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:23.722984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:23.722986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:23.723363 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:23.723563 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:23.723785 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:27:23.723946 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":266,"threads_started":1}
I20260926 07:27:23.727507 4385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:23.728137 4385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:23.734443 4521 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac7828593db48ac8c59bea8543e6928 (DEFAULT_TABLE table=test-table [id=31f356394c0d4e728e7ddb330936dac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.734527 4660 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac7828593db48ac8c59bea8543e6928 (DEFAULT_TABLE table=test-table [id=31f356394c0d4e728e7ddb330936dac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.734707 4799 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac7828593db48ac8c59bea8543e6928 (DEFAULT_TABLE table=test-table [id=31f356394c0d4e728e7ddb330936dac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.734889 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac7828593db48ac8c59bea8543e6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.734951 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac7828593db48ac8c59bea8543e6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.735069 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac7828593db48ac8c59bea8543e6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.740996 4982 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Bootstrap starting.
I20260926 07:27:23.741647 4984 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Bootstrap starting.
I20260926 07:27:23.742473 4983 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Bootstrap starting.
I20260926 07:27:23.742955 4982 tablet_bootstrap.cc:654] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.743669 4984 tablet_bootstrap.cc:654] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.744398 4983 tablet_bootstrap.cc:654] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.745738 4982 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: No bootstrap required, opened a new log
I20260926 07:27:23.745904 4982 ts_tablet_manager.cc:1397] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 07:27:23.746280 4983 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: No bootstrap required, opened a new log
I20260926 07:27:23.746382 4983 ts_tablet_manager.cc:1397] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 07:27:23.746585 4982 raft_consensus.cc:348] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.746805 4982 raft_consensus.cc:374] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.746896 4982 raft_consensus.cc:729] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.746896 4983 raft_consensus.cc:348] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.747094 4983 raft_consensus.cc:374] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.747241 4983 raft_consensus.cc:729] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.747186 4982 consensus_queue.cc:260] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.747483 4983 consensus_queue.cc:260] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.747938 4984 tablet_bootstrap.cc:492] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: No bootstrap required, opened a new log
I20260926 07:27:23.748031 4983 ts_tablet_manager.cc:1428] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:23.748085 4984 ts_tablet_manager.cc:1397] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260926 07:27:23.748975 4984 raft_consensus.cc:348] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.749236 4982 ts_tablet_manager.cc:1428] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260926 07:27:23.749264 4984 raft_consensus.cc:374] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.749393 4984 raft_consensus.cc:729] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5b89da6e641c494f072a972e23df9, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.749676 4984 consensus_queue.cc:260] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.751089 4984 ts_tablet_manager.cc:1428] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:27:23.794545 4988 raft_consensus.cc:482] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:23.794780 4988 raft_consensus.cc:504] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.795555 4988 leader_election.cc:288] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405)
I20260926 07:27:23.796159 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac7828593db48ac8c59bea8543e6928" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9" is_pre_election: true
I20260926 07:27:23.796180 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac7828593db48ac8c59bea8543e6928" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" is_pre_election: true
I20260926 07:27:23.796392 4546 raft_consensus.cc:2454] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 0.
I20260926 07:27:23.796530 4818 raft_consensus.cc:2454] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 0.
I20260926 07:27:23.796923 4611 leader_election.cc:302] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5b89da6e641c494f072a972e23df9, 63d5dd7d367644839d6ea25687ca2308; no voters:
I20260926 07:27:23.797210 4988 raft_consensus.cc:2781] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:23.797340 4988 raft_consensus.cc:482] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:23.797446 4988 raft_consensus.cc:3037] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:23.799639 4988 raft_consensus.cc:504] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.800179 4988 leader_election.cc:288] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 election: Requested vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405)
I20260926 07:27:23.800551 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac7828593db48ac8c59bea8543e6928" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9"
I20260926 07:27:23.800648 4818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac7828593db48ac8c59bea8543e6928" candidate_uuid: "63d5dd7d367644839d6ea25687ca2308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e"
I20260926 07:27:23.800746 4546 raft_consensus.cc:3037] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:23.800858 4818 raft_consensus.cc:3037] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:23.802660 4546 raft_consensus.cc:2454] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 1.
I20260926 07:27:23.803129 4611 leader_election.cc:302] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5b89da6e641c494f072a972e23df9, 63d5dd7d367644839d6ea25687ca2308; no voters:
I20260926 07:27:23.803275 4818 raft_consensus.cc:2454] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d5dd7d367644839d6ea25687ca2308 in term 1.
I20260926 07:27:23.803455 4988 raft_consensus.cc:2781] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:23.803900 4988 raft_consensus.cc:686] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 1 LEADER]: Becoming Leader. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Running, Role: LEADER
I20260926 07:27:23.804221 4988 consensus_queue.cc:237] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.806557 4384 catalog_manager.cc:5495] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "63d5dd7d367644839d6ea25687ca2308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } health_report { overall_health: UNKNOWN } } }
I20260926 07:27:23.829283 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:23.824727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.2.147.194:42909)
0926 07:27:23.825245 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:23.825249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:23.825267 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:23.825314 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:23.825317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:23.825370 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:23.825448 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:23.825745 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:23.826467 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:23.827693 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:23.828075 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:23.828165 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:23.828583 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:23.828679 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:23.828752 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:23.828877 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:27:23.828932 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":297,"threads_started":1}
I20260926 07:27:23.836899 4546 raft_consensus.cc:1261] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Refusing update from remote peer 63d5dd7d367644839d6ea25687ca2308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:23.837023 4818 raft_consensus.cc:1261] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Refusing update from remote peer 63d5dd7d367644839d6ea25687ca2308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:23.837438 4990 consensus_queue.cc:1035] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:23.837728 4988 consensus_queue.cc:1035] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:23.878264 4384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:23.878463 4384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:23.880996 4384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Sending DeleteTablet for 3 replicas of tablet 3ac7828593db48ac8c59bea8543e6928
I20260926 07:27:23.881650 4660 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac7828593db48ac8c59bea8543e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260926 07:27:23.881646 4521 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac7828593db48ac8c59bea8543e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37186
I20260926 07:27:23.881769 4799 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac7828593db48ac8c59bea8543e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:23 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260926 07:27:23.881938 4972 tablet_replica.cc:330] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: stopping tablet replica
I20260926 07:27:23.882030 4973 tablet_replica.cc:330] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: stopping tablet replica
I20260926 07:27:23.882141 4972 raft_consensus.cc:2229] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:23.882210 4973 raft_consensus.cc:2229] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:23.882306 4971 tablet_replica.cc:330] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: stopping tablet replica
I20260926 07:27:23.882366 4973 raft_consensus.cc:2258] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.882431 4972 raft_consensus.cc:2258] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.882504 4971 raft_consensus.cc:2229] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:23.882678 4971 raft_consensus.cc:2258] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:23.882675 4384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:27:23.883311 4972 ts_tablet_manager.cc:1905] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:27:23.883363 4384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:23.883893 4971 ts_tablet_manager.cc:1905] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:23.884060 4973 ts_tablet_manager.cc:1905] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:23.889663 4973 ts_tablet_manager.cc:1918] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:23.889804 4973 log.cc:1196] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/3ac7828593db48ac8c59bea8543e6928
I20260926 07:27:23.890141 4971 ts_tablet_manager.cc:1918] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:23.890233 4973 ts_tablet_manager.cc:1939] T 3ac7828593db48ac8c59bea8543e6928 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting consensus metadata
I20260926 07:27:23.890262 4971 log.cc:1196] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/3ac7828593db48ac8c59bea8543e6928
I20260926 07:27:23.890661 4971 ts_tablet_manager.cc:1939] T 3ac7828593db48ac8c59bea8543e6928 P 04a5b89da6e641c494f072a972e23df9: Deleting consensus metadata
I20260926 07:27:23.890635 4972 ts_tablet_manager.cc:1918] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:27:23.890769 4972 log.cc:1196] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/3ac7828593db48ac8c59bea8543e6928
I20260926 07:27:23.891155 4972 ts_tablet_manager.cc:1939] T 3ac7828593db48ac8c59bea8543e6928 P 63d5dd7d367644839d6ea25687ca2308: Deleting consensus metadata
I20260926 07:27:23.891251 4371 catalog_manager.cc:4841] TS db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405): tablet 3ac7828593db48ac8c59bea8543e6928 (table test-table [id=31f356394c0d4e728e7ddb330936dac9]) successfully deleted
I20260926 07:27:23.891690 4369 catalog_manager.cc:4841] TS 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107): tablet 3ac7828593db48ac8c59bea8543e6928 (table test-table [id=31f356394c0d4e728e7ddb330936dac9]) successfully deleted
I20260926 07:27:23.892241 4370 catalog_manager.cc:4841] TS 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909): tablet 3ac7828593db48ac8c59bea8543e6928 (table test-table [id=31f356394c0d4e728e7ddb330936dac9]) successfully deleted
I20260926 07:27:23.892961 4521 tablet_service.cc:1467] Processing CreateTablet for tablet be27fa420f3d4f1793d60a1d393163e6 (DEFAULT_TABLE table=test-table [id=fe4b84975ed8451dbc2b0d956195cab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.893235 4660 tablet_service.cc:1467] Processing CreateTablet for tablet be27fa420f3d4f1793d60a1d393163e6 (DEFAULT_TABLE table=test-table [id=fe4b84975ed8451dbc2b0d956195cab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.893393 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27fa420f3d4f1793d60a1d393163e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.893503 4799 tablet_service.cc:1467] Processing CreateTablet for tablet be27fa420f3d4f1793d60a1d393163e6 (DEFAULT_TABLE table=test-table [id=fe4b84975ed8451dbc2b0d956195cab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:23.893636 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27fa420f3d4f1793d60a1d393163e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.893858 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be27fa420f3d4f1793d60a1d393163e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:23.897994 4982 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Bootstrap starting.
I20260926 07:27:23.899545 4982 tablet_bootstrap.cc:654] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.900322 4983 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Bootstrap starting.
I20260926 07:27:23.900872 4984 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Bootstrap starting.
I20260926 07:27:23.902281 4983 tablet_bootstrap.cc:654] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.902436 4984 tablet_bootstrap.cc:654] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:23.903203 4982 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: No bootstrap required, opened a new log
I20260926 07:27:23.903481 4982 ts_tablet_manager.cc:1397] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 07:27:23.904364 4982 raft_consensus.cc:348] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.904840 4982 raft_consensus.cc:374] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.904920 4983 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: No bootstrap required, opened a new log
I20260926 07:27:23.904942 4982 raft_consensus.cc:729] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d5dd7d367644839d6ea25687ca2308, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.905055 4983 ts_tablet_manager.cc:1397] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 07:27:23.905181 4982 consensus_queue.cc:260] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.905709 4982 ts_tablet_manager.cc:1428] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:27:23.905767 4983 raft_consensus.cc:348] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.906002 4983 raft_consensus.cc:374] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.906082 4983 raft_consensus.cc:729] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.906303 4983 consensus_queue.cc:260] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.906683 4983 ts_tablet_manager.cc:1428] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 07:27:23.907514 4984 tablet_bootstrap.cc:492] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: No bootstrap required, opened a new log
I20260926 07:27:23.907673 4984 ts_tablet_manager.cc:1397] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:27:23.908349 4984 raft_consensus.cc:348] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.908581 4984 raft_consensus.cc:374] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:23.908656 4984 raft_consensus.cc:729] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a5b89da6e641c494f072a972e23df9, State: Initialized, Role: FOLLOWER
I20260926 07:27:23.908906 4984 consensus_queue.cc:260] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:23.909433 4984 ts_tablet_manager.cc:1428] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:27:24.029305 4911 raft_consensus.cc:482] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:24.029541 4911 raft_consensus.cc:504] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:24.030200 4911 leader_election.cc:288] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:24.030791 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27fa420f3d4f1793d60a1d393163e6" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308" is_pre_election: true
I20260926 07:27:24.030797 4544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27fa420f3d4f1793d60a1d393163e6" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9" is_pre_election: true
I20260926 07:27:24.031018 4544 raft_consensus.cc:2454] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:24.031041 4680 raft_consensus.cc:2454] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 0.
I20260926 07:27:24.031497 4750 leader_election.cc:302] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a5b89da6e641c494f072a972e23df9, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:24.031809 4911 raft_consensus.cc:2781] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:24.032370 4911 raft_consensus.cc:482] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:24.032485 4911 raft_consensus.cc:3037] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:24.034471 4911 raft_consensus.cc:504] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:24.035043 4911 leader_election.cc:288] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Requested vote from peers 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107), 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909)
I20260926 07:27:24.035550 4544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27fa420f3d4f1793d60a1d393163e6" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a5b89da6e641c494f072a972e23df9"
I20260926 07:27:24.035583 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be27fa420f3d4f1793d60a1d393163e6" candidate_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d5dd7d367644839d6ea25687ca2308"
I20260926 07:27:24.035754 4544 raft_consensus.cc:3037] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:24.035794 4680 raft_consensus.cc:3037] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:24.037842 4680 raft_consensus.cc:2454] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:24.037839 4544 raft_consensus.cc:2454] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db8fb9bbdb014c63b017d6fb23bd1d7e in term 1.
I20260926 07:27:24.038323 4751 leader_election.cc:302] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d5dd7d367644839d6ea25687ca2308, db8fb9bbdb014c63b017d6fb23bd1d7e; no voters:
I20260926 07:27:24.038581 4911 raft_consensus.cc:2781] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:24.038755 4911 raft_consensus.cc:686] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 LEADER]: Becoming Leader. State: Replica: db8fb9bbdb014c63b017d6fb23bd1d7e, State: Running, Role: LEADER
I20260926 07:27:24.039005 4911 consensus_queue.cc:237] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } }
I20260926 07:27:24.041477 4384 catalog_manager.cc:5495] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e reported cstate change: term changed from 0 to 1, leader changed from <none> to db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195). New cstate: current_term: 1 leader_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db8fb9bbdb014c63b017d6fb23bd1d7e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35405 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:24.068971 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:24.063986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.2.147.195:35405)
0926 07:27:24.064540 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:24.064544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:24.064562 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:24.064606 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:24.064609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:24.064664 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:24.064744 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:24.065067 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:24.065812 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:24.067530 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:24.067910 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:24.067995 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:24.068390 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:24.068494 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:24.068556 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:24.068692 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:27:24.068746 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":347,"threads_started":1}
I20260926 07:27:24.076908 4544 raft_consensus.cc:1261] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:24.077009 4680 raft_consensus.cc:1261] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Refusing update from remote peer db8fb9bbdb014c63b017d6fb23bd1d7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:24.077450 4965 consensus_queue.cc:1035] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a5b89da6e641c494f072a972e23df9" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:24.077821 4911 consensus_queue.cc:1035] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d5dd7d367644839d6ea25687ca2308" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:27:24.095705 4951 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:27:25.669318 4385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:25.669531 4385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:25.673285 4385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fa46e2424e420eaac4d1b8caed308b: Sending DeleteTablet for 3 replicas of tablet be27fa420f3d4f1793d60a1d393163e6
I20260926 07:27:25.674010 4521 tablet_service.cc:1514] Processing DeleteTablet for tablet be27fa420f3d4f1793d60a1d393163e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:25 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37186
I20260926 07:27:25.674145 4660 tablet_service.cc:1514] Processing DeleteTablet for tablet be27fa420f3d4f1793d60a1d393163e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:25 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260926 07:27:25.674237 4799 tablet_service.cc:1514] Processing DeleteTablet for tablet be27fa420f3d4f1793d60a1d393163e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:25 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260926 07:27:25.675170 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4456
I20260926 07:27:25.686584 5026 tablet_replica.cc:330] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: stopping tablet replica
I20260926 07:27:25.687024 5026 raft_consensus.cc:2229] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:25.687492 5026 raft_consensus.cc:2258] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:25.688144 5028 tablet_replica.cc:330] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: stopping tablet replica
I20260926 07:27:25.688354 5028 raft_consensus.cc:2229] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:25.688664 5028 raft_consensus.cc:2258] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:25.689394 5026 ts_tablet_manager.cc:1905] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:25.691799 5028 ts_tablet_manager.cc:1905] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:25.695860 5026 ts_tablet_manager.cc:1918] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:25.695986 5026 log.cc:1196] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/be27fa420f3d4f1793d60a1d393163e6
I20260926 07:27:25.696439 5026 ts_tablet_manager.cc:1939] T be27fa420f3d4f1793d60a1d393163e6 P 04a5b89da6e641c494f072a972e23df9: Deleting consensus metadata
I20260926 07:27:25.698500 5028 ts_tablet_manager.cc:1918] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:25.698657 5028 log.cc:1196] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/be27fa420f3d4f1793d60a1d393163e6
I20260926 07:27:25.699080 5028 ts_tablet_manager.cc:1939] T be27fa420f3d4f1793d60a1d393163e6 P db8fb9bbdb014c63b017d6fb23bd1d7e: Deleting consensus metadata
I20260926 07:27:25.700055 4371 catalog_manager.cc:4841] TS db8fb9bbdb014c63b017d6fb23bd1d7e (127.2.147.195:35405): tablet be27fa420f3d4f1793d60a1d393163e6 (table test-table [id=fe4b84975ed8451dbc2b0d956195cab9]) successfully deleted
I20260926 07:27:25.702482 4369 catalog_manager.cc:4841] TS 04a5b89da6e641c494f072a972e23df9 (127.2.147.193:38107): tablet be27fa420f3d4f1793d60a1d393163e6 (table test-table [id=fe4b84975ed8451dbc2b0d956195cab9]) successfully deleted
I20260926 07:27:25.702801 5027 tablet_replica.cc:330] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: stopping tablet replica
I20260926 07:27:25.703023 5027 raft_consensus.cc:2229] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:25.703323 5027 raft_consensus.cc:2258] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:25.704370 5027 ts_tablet_manager.cc:1905] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:25.709066 5027 ts_tablet_manager.cc:1918] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:27:25.709189 5027 log.cc:1196] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/be27fa420f3d4f1793d60a1d393163e6
I20260926 07:27:25.709532 5027 ts_tablet_manager.cc:1939] T be27fa420f3d4f1793d60a1d393163e6 P 63d5dd7d367644839d6ea25687ca2308: Deleting consensus metadata
I20260926 07:27:25.710492 4370 catalog_manager.cc:4841] TS 63d5dd7d367644839d6ea25687ca2308 (127.2.147.194:42909): tablet be27fa420f3d4f1793d60a1d393163e6 (table test-table [id=fe4b84975ed8451dbc2b0d956195cab9]) successfully deleted
Sep 26 07:27:25 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:25.720176 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:25.677707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.2.147.193:38107)
0926 07:27:25.701333 (+ 23626us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:25.701342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:25.701371 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:27:25.701440 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:25.701447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:25.701562 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:25.701687 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:25.703361 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.704515 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.705765 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.706353 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.706471 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:25.714496 (+ 8025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:25.714510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:25.714527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:25.714557 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:25.717464 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.718027 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.718034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.718036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.718162 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.718444 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.718447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.718449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.718757 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:25.718951 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:25.719679 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 07:27:25.719825 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23448,"thread_start_us":24454,"threads_started":1}
W20260926 07:27:25.720769 4586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37198
W20260926 07:27:25.721294 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4456: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:25.721499 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 4456
I20260926 07:27:25.738915 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4595
Sep 26 07:27:25 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:25.752875 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:25.739638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.2.147.194:42909)
0926 07:27:25.740104 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:25.740108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:25.740124 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:27:25.740175 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:25.740178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:25.740249 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:25.740336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:25.741593 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.742275 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.743456 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.743817 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.743905 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:25.747789 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:25.747803 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:25.747819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:25.747849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:25.750504 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.751037 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.751064 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.751070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.751314 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.751607 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.751610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.751612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.751937 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:25.752142 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:25.752469 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 07:27:25.752621 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":258,"threads_started":1}
W20260926 07:27:25.753499 4725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34360
W20260926 07:27:25.754036 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4595: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:25.754154 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 4595
I20260926 07:27:25.776183 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4734
Sep 26 07:27:25 dist-test-slave-1dwd krb5kdc[4340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:25.790227 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:25.776987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.2.147.195:35405)
0926 07:27:25.777444 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:25.777449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:25.777464 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:27:25.777514 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:25.777517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:25.777581 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:25.777742 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:25.778876 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.779738 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.780840 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:25.781257 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:25.781338 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:25.785142 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:25.785155 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:25.785171 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:25.785216 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:25.788123 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.788677 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.788683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.788686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.788800 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:25.789088 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:25.789091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:25.789092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:25.789383 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:25.789614 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:25.789801 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:27:25.789981 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
W20260926 07:27:25.790931 4864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59710
W20260926 07:27:25.791498 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:25.791633 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 4734
I20260926 07:27:25.816084 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 4353
I20260926 07:27:25.817144 4425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:25.930335 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 4353
2026-09-26T07:27:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8161 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:26 dist-test-slave-1dwd krb5kdc[5036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:26 dist-test-slave-1dwd krb5kdc[5036](info): set up 2 sockets
Sep 26 07:27:26 dist-test-slave-1dwd krb5kdc[5036](info): commencing operation
krb5kdc: starting...
W20260926 07:27:27.003616 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:27 dist-test-slave-1dwd krb5kdc[5036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:27.188853 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:42613
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:27.505060 5049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:27.505329 5049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:27.505487 5049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:27.512382 5049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:27.512481 5049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:27.512569 5049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:27.512660 5049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:27.512723 5049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:27.522507 5049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:42613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:27.523914 5049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:27.525306 5049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:27.536936 5054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:27.536914 5055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:27.537228 5057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:27.537686 5049 server_base.cc:1034] running on GCE node
I20260926 07:27:27.538467 5049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:27.539924 5049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:27.541109 5049 hybrid_clock.cc:648] HybridClock initialized: now 1790407647541053 us; error 45 us; skew 500 ppm
I20260926 07:27:27.541445 5049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:27 dist-test-slave-1dwd krb5kdc[5036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407647, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:27.544440 5049 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:27.545972 5049 webserver.cc:460] Webserver started at http://127.2.147.254:46111/ using document root <none> and password file <none>
I20260926 07:27:27.546638 5049 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:27.546782 5049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:27.547080 5049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:27.549903 5049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "f8836e5d48274205a67bece90bac4820"
format_stamp: "Formatted at 2026-09-26 07:27:27 on dist-test-slave-1dwd"
server_key: "8a31ff2851734d560002fa401708f95d"
server_key_iv: "732f247d516205ed1ab8035c08c1d412"
server_key_version: "encryptionkey@0"
I20260926 07:27:27.550678 5049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "f8836e5d48274205a67bece90bac4820"
format_stamp: "Formatted at 2026-09-26 07:27:27 on dist-test-slave-1dwd"
server_key: "8a31ff2851734d560002fa401708f95d"
server_key_iv: "732f247d516205ed1ab8035c08c1d412"
server_key_version: "encryptionkey@0"
I20260926 07:27:27.555580 5049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:27:27.558733 5064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:27.559930 5049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:27:27.560109 5049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f8836e5d48274205a67bece90bac4820"
format_stamp: "Formatted at 2026-09-26 07:27:27 on dist-test-slave-1dwd"
server_key: "8a31ff2851734d560002fa401708f95d"
server_key_iv: "732f247d516205ed1ab8035c08c1d412"
server_key_version: "encryptionkey@0"
I20260926 07:27:27.560303 5049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:27.577797 5049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:27.581645 5049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:27.581957 5049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:27.599815 5049 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:42613
I20260926 07:27:27.599833 5115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:42613 every 8 connection(s)
I20260926 07:27:27.601157 5049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:27.604811 5116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:27.606135 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5049
I20260926 07:27:27.606345 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:27.606891 2639 external_mini_cluster.cc:1428] Setting key a01bd5027b59677c2a28d06a3d22d377
I20260926 07:27:27.618090 5116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: Bootstrap starting.
Sep 26 07:27:27 dist-test-slave-1dwd krb5kdc[5036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:27.621285 5116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:27.622394 5116 log.cc:824] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:27.625260 5116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: No bootstrap required, opened a new log
I20260926 07:27:27.628984 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:27.608822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.2.147.254:42613)
0926 07:27:27.609583 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:27.609594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:27.609635 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:27:27.611124 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:27.611148 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:27.611589 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:27.611882 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:27.613619 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:27.614930 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:27.616769 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:27.617405 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:27.617529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:27.621847 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:27.621854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:27.621868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:27.621900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:27.625145 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:27.625909 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:27.625914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:27.625919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:27.626070 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:27.626473 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:27.626476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:27.626477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:27.626940 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:27.627167 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:27.627645 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 07:27:27.627988 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":365,"threads_started":1}
I20260926 07:27:27.633167 5116 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } }
I20260926 07:27:27.633507 5116 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:27.633616 5116 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8836e5d48274205a67bece90bac4820, State: Initialized, Role: FOLLOWER
I20260926 07:27:27.634191 5116 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } }
I20260926 07:27:27.634428 5116 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:27.634583 5116 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:27.634764 5116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:27.636807 5116 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } }
I20260926 07:27:27.637253 5116 leader_election.cc:302] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8836e5d48274205a67bece90bac4820; no voters:
I20260926 07:27:27.637918 5116 leader_election.cc:288] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:27.638013 5121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:27.638653 5121 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [term 1 LEADER]: Becoming Leader. State: Replica: f8836e5d48274205a67bece90bac4820, State: Running, Role: LEADER
I20260926 07:27:27.639240 5121 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } }
I20260926 07:27:27.640005 5116 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:27.642102 5122 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8836e5d48274205a67bece90bac4820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } } }
I20260926 07:27:27.642498 5122 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:27.642241 5123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8836e5d48274205a67bece90bac4820. Latest consensus state: current_term: 1 leader_uuid: "f8836e5d48274205a67bece90bac4820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8836e5d48274205a67bece90bac4820" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42613 } } }
I20260926 07:27:27.642637 5123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820 [sys.catalog]: This master's current role is: LEADER
W20260926 07:27:27.650888 5137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:27.651021 5137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:27.651162 5127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:27.652546 5127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:27.660446 5127 catalog_manager.cc:1347] Generated new cluster ID: a726f0ac014f43b7909382a646391ac3
I20260926 07:27:27.660548 5127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:27.688656 5127 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:27.690495 5127 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:27.698035 5127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8836e5d48274205a67bece90bac4820: Generated new TSK 0
I20260926 07:27:27.698861 5127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:27.752354 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:28.074278 5142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:28.074543 5142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:28.074635 5142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:28.081537 5142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:28.081631 5142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:28.081739 5142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:28.091128 5142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:28.092329 5142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:28.093561 5142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:28.105964 5147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:28.106045 5150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:28.105986 5148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:28.106866 5142 server_base.cc:1034] running on GCE node
I20260926 07:27:28.107532 5142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:28.108440 5142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:28.109619 5142 hybrid_clock.cc:648] HybridClock initialized: now 1790407648109564 us; error 48 us; skew 500 ppm
I20260926 07:27:28.109926 5142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:28 dist-test-slave-1dwd krb5kdc[5036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407648, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:28.113011 5142 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:28.114568 5142 webserver.cc:460] Webserver started at http://127.2.147.193:40185/ using document root <none> and password file <none>
I20260926 07:27:28.115299 5142 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:28.115437 5142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:28.115748 5142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:28.118525 5142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "f84d940bae7f475ba0a3c2bda4af983c"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "7e1a267d410a8eed7f8b699d42e9ea09"
server_key_iv: "435709683d6d9aa2bfb8510dfe064a04"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.119330 5142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "f84d940bae7f475ba0a3c2bda4af983c"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "7e1a267d410a8eed7f8b699d42e9ea09"
server_key_iv: "435709683d6d9aa2bfb8510dfe064a04"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.124357 5142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:27:28.127676 5157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.129315 5142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:28.129484 5142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "f84d940bae7f475ba0a3c2bda4af983c"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "7e1a267d410a8eed7f8b699d42e9ea09"
server_key_iv: "435709683d6d9aa2bfb8510dfe064a04"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.129673 5142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:28.155853 5142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:28.159631 5142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:28.159907 5142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:28.160759 5142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:28.161966 5142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:28.162065 5142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.162215 5142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:28.162285 5142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.194576 5142 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45019
I20260926 07:27:28.194631 5269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45019 every 8 connection(s)
I20260926 07:27:28.195832 5142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:28.199934 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5142
I20260926 07:27:28.200133 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:28.200727 2639 external_mini_cluster.cc:1428] Setting key 54300c576b20a4c755a143b768c3c023
Sep 26 07:27:28 dist-test-slave-1dwd krb5kdc[5036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407648, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:28.215009 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:28.198311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:54275 (local address 127.2.147.254:42613)
0926 07:27:28.198726 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:28.198731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:28.199805 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 07:27:28.201814 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:28.201820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:28.201913 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:28.202026 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:28.203603 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.204269 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.205800 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.206219 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.206343 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:28.209583 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:28.209588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:28.209591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:28.209621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:28.212124 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:28.212805 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:28.212817 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:28.212820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:28.212927 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:28.213364 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:28.213369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:28.213372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:28.213626 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:28.213790 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:28.214437 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 07:27:28.214664 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":191,"threads_started":1}
I20260926 07:27:28.215962 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:28.198920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:42613 (local address 127.2.147.193:54275)
0926 07:27:28.199653 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:28.199688 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:28.201510 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:28.202287 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:28.202300 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:28.202860 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:28.203450 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:28.203468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.204435 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.204442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:28.205660 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:28.205675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.206426 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.206432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:28.206650 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:28.207332 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:28.207352 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:28.209373 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:28.212264 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:28.212272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:28.212288 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:28.212681 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:28.213046 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:28.213052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:28.213056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:28.213246 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:28.213760 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:28.213768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:28.214075 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 07:27:28.214330 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 07:27:28.214828 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:28.217561 5270 heartbeater.cc:347] Connected to a master server at 127.2.147.254:42613
I20260926 07:27:28.217868 5270 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:28.218598 5270 heartbeater.cc:511] Master 127.2.147.254:42613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:27:28.220754 5081 ts_manager.cc:194] Registered new tserver with Master: f84d940bae7f475ba0a3c2bda4af983c (127.2.147.193:45019)
I20260926 07:27:28.222131 5081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:54275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:28.240515 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:28.554173 5276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:28.554436 5276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:28.554617 5276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:28.561668 5276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:28.561789 5276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:28.561962 5276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:28.571609 5276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:28.572803 5276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:28.574069 5276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:28.586956 5284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:28.586908 5282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:28.586994 5281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:28.587692 5276 server_base.cc:1034] running on GCE node
I20260926 07:27:28.588270 5276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:28.589140 5276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:28.590368 5276 hybrid_clock.cc:648] HybridClock initialized: now 1790407648590316 us; error 63 us; skew 500 ppm
I20260926 07:27:28.590673 5276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:28 dist-test-slave-1dwd krb5kdc[5036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407648, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:28.593766 5276 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:28.595166 5276 webserver.cc:460] Webserver started at http://127.2.147.194:46473/ using document root <none> and password file <none>
I20260926 07:27:28.595947 5276 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:28.596088 5276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:28.596395 5276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:28.599334 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "8b16ea53e33b426aa71603c761b682ef"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "a970a9f6d3b334687a8b7182dd6c39fe"
server_key_iv: "087830024735dd808135c2612b38404e"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.600319 5276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "8b16ea53e33b426aa71603c761b682ef"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "a970a9f6d3b334687a8b7182dd6c39fe"
server_key_iv: "087830024735dd808135c2612b38404e"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.606380 5276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 07:27:28.610363 5291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.611801 5276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:28.611944 5276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "8b16ea53e33b426aa71603c761b682ef"
format_stamp: "Formatted at 2026-09-26 07:27:28 on dist-test-slave-1dwd"
server_key: "a970a9f6d3b334687a8b7182dd6c39fe"
server_key_iv: "087830024735dd808135c2612b38404e"
server_key_version: "encryptionkey@0"
I20260926 07:27:28.612139 5276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:28.629710 5276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:28.633443 5276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:28.633661 5276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:28.634438 5276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:28.635661 5276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:28.635730 5276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.635810 5276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:28.635854 5276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:28.668545 5276 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:40229
I20260926 07:27:28.668586 5403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:40229 every 8 connection(s)
I20260926 07:27:28.669848 5276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:28.677114 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5276
I20260926 07:27:28.677322 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:28.677726 2639 external_mini_cluster.cc:1428] Setting key 835a83dcf9991e4250a15ba8f74613d4
Sep 26 07:27:28 dist-test-slave-1dwd krb5kdc[5036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407648, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:28.690026 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:28.672253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55999 (local address 127.2.147.254:42613)
0926 07:27:28.672413 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:28.672418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:28.673635 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 07:27:28.675484 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:28.675488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:28.675553 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:28.675641 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:28.677168 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.678013 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.679879 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.680252 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.680344 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:28.684535 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:28.684539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:28.684543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:28.684573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:28.687140 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:28.687883 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:28.687889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:28.687892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:28.688013 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:28.688429 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:28.688433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:28.688437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:28.688696 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:28.688855 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:28.689485 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 07:27:28.689703 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:27:28.690932 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:28.672791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:42613 (local address 127.2.147.194:55999)
0926 07:27:28.673469 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:28.673509 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:28.675179 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:28.675842 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:28.675850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:28.676335 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:28.677004 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:28.677027 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.678159 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.678164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:28.679742 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:28.679757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:28.680455 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:28.680464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:28.680810 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:28.681691 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:28.681738 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:28.684329 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:28.687338 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:28.687346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:28.687362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:28.687760 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:28.688119 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:28.688125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:28.688128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:28.688314 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:28.688819 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:28.688826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:28.689150 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 07:27:28.689398 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 07:27:28.689889 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260926 07:27:28.692350 5404 heartbeater.cc:347] Connected to a master server at 127.2.147.254:42613
I20260926 07:27:28.692653 5404 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:28.693351 5404 heartbeater.cc:511] Master 127.2.147.254:42613 requested a full tablet report, sending...
I20260926 07:27:28.694775 5081 ts_manager.cc:194] Registered new tserver with Master: 8b16ea53e33b426aa71603c761b682ef (127.2.147.194:40229)
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:27:28.695748 5081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:55999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:28.719321 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:29.031314 5409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:29.031584 5409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:29.031756 5409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:29.039005 5409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:29.039134 5409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:29.039333 5409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:29.049019 5409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:29.050331 5409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:29.051645 5409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:29.064486 5414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:29.064355 5417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:29.064386 5415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:29.065166 5409 server_base.cc:1034] running on GCE node
I20260926 07:27:29.065644 5409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:29.066464 5409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:29.067639 5409 hybrid_clock.cc:648] HybridClock initialized: now 1790407649067583 us; error 50 us; skew 500 ppm
I20260926 07:27:29.067879 5409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:29 dist-test-slave-1dwd krb5kdc[5036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:29.070712 5409 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:29.072095 5409 webserver.cc:460] Webserver started at http://127.2.147.195:41785/ using document root <none> and password file <none>
I20260926 07:27:29.072728 5409 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:29.072866 5409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:29.073170 5409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:29.076038 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "a47014a859604a98ab427be4cc026263"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-1dwd"
server_key: "f367d16c53eef0f773e7c3e704142165"
server_key_iv: "f71c0040f2d6537904fb54cfaaec0395"
server_key_version: "encryptionkey@0"
I20260926 07:27:29.076848 5409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "a47014a859604a98ab427be4cc026263"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-1dwd"
server_key: "f367d16c53eef0f773e7c3e704142165"
server_key_iv: "f71c0040f2d6537904fb54cfaaec0395"
server_key_version: "encryptionkey@0"
I20260926 07:27:29.081602 5409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:29.084898 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:29.086205 5409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:29.086370 5409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "a47014a859604a98ab427be4cc026263"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-1dwd"
server_key: "f367d16c53eef0f773e7c3e704142165"
server_key_iv: "f71c0040f2d6537904fb54cfaaec0395"
server_key_version: "encryptionkey@0"
I20260926 07:27:29.086565 5409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:29.115531 5409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:29.119794 5409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:29.120090 5409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:29.120985 5409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:29.122337 5409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:29.122407 5409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:29.122571 5409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:29.122622 5409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:29.154297 5409 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:46843
I20260926 07:27:29.154331 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:46843 every 8 connection(s)
I20260926 07:27:29.155610 5409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:29.155925 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5409
I20260926 07:27:29.156134 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:29.156749 2639 external_mini_cluster.cc:1428] Setting key d94dfb4679c4dadd59cde9cd2e3e0b4f
Sep 26 07:27:29 dist-test-slave-1dwd krb5kdc[5036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:29.175598 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:29.158621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:51303 (local address 127.2.147.254:42613)
0926 07:27:29.158804 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:29.158810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:29.160162 (+ 1352us) server_negotiation.cc:407] Connection header received
0926 07:27:29.161896 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:29.161901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:29.161982 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:29.162097 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:29.163628 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.164324 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.165895 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.166312 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.166442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:29.169700 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:29.169705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:29.169712 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:29.169747 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:29.172756 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:29.173450 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:29.173455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:29.173459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:29.173574 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:29.173981 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:29.173985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:29.173989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:29.174255 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:29.174403 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:29.175027 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 07:27:29.175259 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:29.176491 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:29.159163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:42613 (local address 127.2.147.195:51303)
0926 07:27:29.160021 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:29.160063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:29.161647 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:29.162323 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:29.162331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:29.162824 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:29.163464 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:29.163490 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.164463 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.164468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:29.165752 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:29.165767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.166519 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.166528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:29.166762 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:29.167460 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:29.167481 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:29.169512 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:29.172889 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:29.172898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:29.172912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:29.173327 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:29.173682 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:29.173687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:29.173691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:29.173865 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:29.174377 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:29.174384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:29.174699 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 07:27:29.174944 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 07:27:29.175449 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":302,"threads_started":1}
I20260926 07:27:29.177845 5537 heartbeater.cc:347] Connected to a master server at 127.2.147.254:42613
I20260926 07:27:29.178126 5537 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:29.178776 5537 heartbeater.cc:511] Master 127.2.147.254:42613 requested a full tablet report, sending...
I20260926 07:27:29.180248 5081 ts_manager.cc:194] Registered new tserver with Master: a47014a859604a98ab427be4cc026263 (127.2.147.195:46843)
I20260926 07:27:29.180999 5081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:51303
I20260926 07:27:29.184149 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:27:29.214828 5272 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:29.212380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.2.147.254:42613)
0926 07:27:29.212549 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:29.212554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:29.213092 (+ 538us) server_negotiation.cc:407] Connection header received
0926 07:27:29.213245 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:29.213248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:29.213310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:29.213412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:29.214589 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50690: BlockingRecv error: recv got EOF from 127.0.0.1:50690 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:29.218016 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5142
W20260926 07:27:29.220607 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5142: Not authorized: Client connection negotiation failed: client connection to 127.2.147.193:45019: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:27:29.220795 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5142
I20260926 07:27:29.233297 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5276
W20260926 07:27:29.235723 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5276: Not authorized: Client connection negotiation failed: client connection to 127.2.147.194:40229: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:27:29.235882 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5276
I20260926 07:27:29.251166 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5409
W20260926 07:27:29.253942 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5409: Not authorized: Client connection negotiation failed: client connection to 127.2.147.195:46843: server requires authentication, but client does not have Kerberos credentials available
I20260926 07:27:29.254076 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5409
I20260926 07:27:29.269306 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5049
I20260926 07:27:29.270457 5111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:29.374285 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5049
2026-09-26T07:27:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3441 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:29 dist-test-slave-1dwd krb5kdc[5554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:29 dist-test-slave-1dwd krb5kdc[5554](info): set up 2 sockets
Sep 26 07:27:29 dist-test-slave-1dwd krb5kdc[5554](info): commencing operation
krb5kdc: starting...
W20260926 07:27:30.459566 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:30 dist-test-slave-1dwd krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:30.643023 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:42299
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:30.961637 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:30.961925 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:30.962014 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:30.969163 5567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:30.969259 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:30.969400 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:30.969496 5567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:30.969578 5567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:30.978938 5567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:42299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:30.980206 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:30.981462 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:30.992738 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.992689 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.992767 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:30.993525 5567 server_base.cc:1034] running on GCE node
I20260926 07:27:30.994191 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:30.995515 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:30.996729 5567 hybrid_clock.cc:648] HybridClock initialized: now 1790407650996690 us; error 53 us; skew 500 ppm
I20260926 07:27:30.997054 5567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:30 dist-test-slave-1dwd krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:31.004683 5567 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:31.006093 5567 webserver.cc:460] Webserver started at http://127.2.147.254:34889/ using document root <none> and password file <none>
I20260926 07:27:31.007068 5567 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:31.007238 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:31.007548 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:31.010380 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "ed121b12e5874fd6be2c15d5c0c5e09c"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-1dwd"
server_key: "f0c8ee31a9e3415b8e0a6d444eed5ab6"
server_key_iv: "6fc0d3e25ea8e43b5750b4c73229d0ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:31.011148 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "ed121b12e5874fd6be2c15d5c0c5e09c"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-1dwd"
server_key: "f0c8ee31a9e3415b8e0a6d444eed5ab6"
server_key_iv: "6fc0d3e25ea8e43b5750b4c73229d0ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:31.016487 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:27:31.019774 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.020998 5567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:27:31.021168 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "ed121b12e5874fd6be2c15d5c0c5e09c"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-1dwd"
server_key: "f0c8ee31a9e3415b8e0a6d444eed5ab6"
server_key_iv: "6fc0d3e25ea8e43b5750b4c73229d0ff"
server_key_version: "encryptionkey@0"
I20260926 07:27:31.021349 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:31.050316 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:31.054718 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:31.055105 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:31.072276 5567 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:42299
I20260926 07:27:31.072319 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:42299 every 8 connection(s)
I20260926 07:27:31.073609 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:31.077140 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:31.080104 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5567
I20260926 07:27:31.080307 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:31.081110 2639 external_mini_cluster.cc:1428] Setting key dae2c41b83c96b71a420476e64c7709c
I20260926 07:27:31.089609 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c: Bootstrap starting.
I20260926 07:27:31.093456 5634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:31.094805 5634 log.cc:824] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:27:31 dist-test-slave-1dwd krb5kdc[5554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:31.097507 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c: No bootstrap required, opened a new log
I20260926 07:27:31.104103 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.083068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.2.147.254:42299)
0926 07:27:31.083890 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:31.083901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:31.083940 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:31.085526 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:31.085547 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:31.085930 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:31.086259 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:31.088031 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.089298 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.091033 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.091771 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.091892 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.096443 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:31.096451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:31.096465 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:31.096497 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:31.099545 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.100229 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.100235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.100241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.100395 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.100773 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.100778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.100781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.101329 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:31.101648 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:31.102086 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 07:27:31.102577 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":429,"threads_started":1}
I20260926 07:27:31.106925 5634 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } }
I20260926 07:27:31.107340 5634 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:31.107435 5634 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed121b12e5874fd6be2c15d5c0c5e09c, State: Initialized, Role: FOLLOWER
I20260926 07:27:31.108103 5634 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } }
I20260926 07:27:31.108368 5634 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:31.108495 5634 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:31.108650 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:31.108670 5634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:31.111673 5634 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } }
I20260926 07:27:31.112231 5634 leader_election.cc:302] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed121b12e5874fd6be2c15d5c0c5e09c; no voters:
I20260926 07:27:31.112839 5634 leader_election.cc:288] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:31.112943 5639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:31.114964 5634 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:31.116671 5639 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [term 1 LEADER]: Becoming Leader. State: Replica: ed121b12e5874fd6be2c15d5c0c5e09c, State: Running, Role: LEADER
I20260926 07:27:31.117156 5639 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } }
I20260926 07:27:31.119819 5640 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } } }
I20260926 07:27:31.120239 5640 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:31.123425 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed121b12e5874fd6be2c15d5c0c5e09c. Latest consensus state: current_term: 1 leader_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed121b12e5874fd6be2c15d5c0c5e09c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 42299 } } }
I20260926 07:27:31.123739 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:31.124046 5646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:31.131420 5646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:31.157055 5646 catalog_manager.cc:1347] Generated new cluster ID: 422c577085234ce192a85fc6fd369602
I20260926 07:27:31.157205 5646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:31.177147 5646 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:31.179464 5646 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:31.196071 5646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed121b12e5874fd6be2c15d5c0c5e09c: Generated new TSK 0
I20260926 07:27:31.197165 5646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:31.208128 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.200761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.2.147.254:42299)
0926 07:27:31.200939 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:31.200947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:31.201374 (+ 427us) server_negotiation.cc:407] Connection header received
0926 07:27:31.201585 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:31.201590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:31.201667 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:31.201778 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:31.202764 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.203603 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.204901 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.205387 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.205533 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.206640 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:31.206647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:31.206651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:31.207319 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:31.207378 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:31.207404 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:31.207557 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:31.207702 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:27:31.207792 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:31.214865 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5567
I20260926 07:27:31.216181 5629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:31.307791 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5567
2026-09-26T07:27:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1927 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:31 dist-test-slave-1dwd krb5kdc[5668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:31 dist-test-slave-1dwd krb5kdc[5668](info): set up 2 sockets
Sep 26 07:27:31 dist-test-slave-1dwd krb5kdc[5668](info): commencing operation
krb5kdc: starting...
W20260926 07:27:32.375607 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:32 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:32.558790 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39319
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:32.870417 5681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:32.870782 5681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:32.870929 5681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:32.878124 5681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:32.878264 5681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:32.878371 5681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:32.878468 5681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:32.878541 5681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:32.888263 5681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:32.889547 5681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:32.890885 5681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:32.902443 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:32.902400 5689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:32.902467 5687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:32.903070 5681 server_base.cc:1034] running on GCE node
I20260926 07:27:32.903847 5681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:32.905165 5681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:32.906388 5681 hybrid_clock.cc:648] HybridClock initialized: now 1790407652906332 us; error 54 us; skew 500 ppm
I20260926 07:27:32.906714 5681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:32 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:32.909773 5681 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:32.911113 5681 webserver.cc:460] Webserver started at http://127.2.147.254:45421/ using document root <none> and password file <none>
I20260926 07:27:32.911789 5681 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:32.911932 5681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:32.912251 5681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:32.915150 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "08c848a4226340d49eff096adb672e03"
format_stamp: "Formatted at 2026-09-26 07:27:32 on dist-test-slave-1dwd"
server_key: "b33f092d3823adcc3c929ec8b7e0b7c1"
server_key_iv: "619042302117eff117c4acf892619c6e"
server_key_version: "encryptionkey@0"
I20260926 07:27:32.915966 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "08c848a4226340d49eff096adb672e03"
format_stamp: "Formatted at 2026-09-26 07:27:32 on dist-test-slave-1dwd"
server_key: "b33f092d3823adcc3c929ec8b7e0b7c1"
server_key_iv: "619042302117eff117c4acf892619c6e"
server_key_version: "encryptionkey@0"
I20260926 07:27:32.921224 5681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:27:32.924757 5696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.925993 5681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:32.926167 5681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "08c848a4226340d49eff096adb672e03"
format_stamp: "Formatted at 2026-09-26 07:27:32 on dist-test-slave-1dwd"
server_key: "b33f092d3823adcc3c929ec8b7e0b7c1"
server_key_iv: "619042302117eff117c4acf892619c6e"
server_key_version: "encryptionkey@0"
I20260926 07:27:32.926362 5681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:32.951423 5681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:32.955675 5681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:32.955958 5681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:32.973518 5681 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:39319
I20260926 07:27:32.973551 5747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:39319 every 8 connection(s)
I20260926 07:27:32.974833 5681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:32.976912 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5681
I20260926 07:27:32.977124 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:32.977900 2639 external_mini_cluster.cc:1428] Setting key 99152307120987e616b8b4e29dca9deb
I20260926 07:27:32.979300 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:27:32 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:32.992945 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03: Bootstrap starting.
I20260926 07:27:32.996994 5748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:32.998237 5748 log.cc:824] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:33.000751 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.979920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.2.147.254:39319)
0926 07:27:32.980693 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:32.980706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:32.980745 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:32.982176 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:32.982196 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:32.982597 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:32.982920 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:32.984532 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.985819 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.987514 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.988160 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.988295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.992736 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:32.992747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:32.992774 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:32.992817 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:32.996542 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.997138 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.997146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.997156 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.997373 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.997710 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.997715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.997718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.998374 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:32.998687 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:32.999066 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 07:27:32.999507 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":349,"threads_started":1}
I20260926 07:27:33.001602 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03: No bootstrap required, opened a new log
I20260926 07:27:33.009292 5748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } }
I20260926 07:27:33.009591 5748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:33.009660 5748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c848a4226340d49eff096adb672e03, State: Initialized, Role: FOLLOWER
I20260926 07:27:33.010103 5748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } }
I20260926 07:27:33.010295 5748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:33.010389 5748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:33.010509 5748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:33.012504 5748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } }
I20260926 07:27:33.012892 5748 leader_election.cc:302] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c848a4226340d49eff096adb672e03; no voters:
I20260926 07:27:33.013432 5748 leader_election.cc:288] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:33.013603 5753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:33.014278 5753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [term 1 LEADER]: Becoming Leader. State: Replica: 08c848a4226340d49eff096adb672e03, State: Running, Role: LEADER
I20260926 07:27:33.014809 5753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } }
I20260926 07:27:33.015590 5748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:33.018002 5755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c848a4226340d49eff096adb672e03. Latest consensus state: current_term: 1 leader_uuid: "08c848a4226340d49eff096adb672e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } } }
I20260926 07:27:33.018386 5755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:33.019780 5762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:33.019742 5754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c848a4226340d49eff096adb672e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c848a4226340d49eff096adb672e03" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39319 } } }
I20260926 07:27:33.020085 5754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:33.024272 5762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:33.032259 5762 catalog_manager.cc:1347] Generated new cluster ID: 11ad141d286c40759cddcbae5398e532
I20260926 07:27:33.032385 5762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:33.047079 5762 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:33.048383 5762 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:33.059808 5762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c848a4226340d49eff096adb672e03: Generated new TSK 0
I20260926 07:27:33.060612 5762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:33.114580 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:33.428253 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:33.428510 5774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:33.428656 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:33.435990 5774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:33.436120 5774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:33.436303 5774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:33.445328 5774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:33.446589 5774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:33.447793 5774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:33.460794 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:33.460948 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:33.460898 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:33.461532 5774 server_base.cc:1034] running on GCE node
I20260926 07:27:33.462176 5774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:33.463088 5774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:33.464315 5774 hybrid_clock.cc:648] HybridClock initialized: now 1790407653464283 us; error 67 us; skew 500 ppm
I20260926 07:27:33.464629 5774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:33 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407653, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:33.471813 5774 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:33.473285 5774 webserver.cc:460] Webserver started at http://127.2.147.193:43753/ using document root <none> and password file <none>
I20260926 07:27:33.473968 5774 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:33.474109 5774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:33.474406 5774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:33.477253 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "77b4143adc334b3f85bcbacaf3bb5133"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "16eec5c1c932ff358d5e4fb2f0cd9f39"
server_key_iv: "5709fa5b5aac4fd3b5fb8ae4954e2fcd"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.477984 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "77b4143adc334b3f85bcbacaf3bb5133"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "16eec5c1c932ff358d5e4fb2f0cd9f39"
server_key_iv: "5709fa5b5aac4fd3b5fb8ae4954e2fcd"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.483031 5774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 07:27:33.486379 5789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:33.487744 5774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:33.487931 5774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "77b4143adc334b3f85bcbacaf3bb5133"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "16eec5c1c932ff358d5e4fb2f0cd9f39"
server_key_iv: "5709fa5b5aac4fd3b5fb8ae4954e2fcd"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.488130 5774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:33.510468 5774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:33.516696 5774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:33.517117 5774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:33.518447 5774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:33.520383 5774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:33.520480 5774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:33.520610 5774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:33.520673 5774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:33.557559 5774 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38077
I20260926 07:27:33.557581 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38077 every 8 connection(s)
I20260926 07:27:33.558895 5774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:33.565384 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5774
I20260926 07:27:33.565593 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:33.566226 2639 external_mini_cluster.cc:1428] Setting key 3cc4efebe318d51fa7746598dae7b513
Sep 26 07:27:33 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407653, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:33.579173 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.561758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:45739 (local address 127.2.147.254:39319)
0926 07:27:33.562190 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.562195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.563197 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 07:27:33.565020 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.565032 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.565148 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.565265 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:33.566876 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.568026 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.569561 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.570268 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.570405 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.573797 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:33.573804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:33.573808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:33.573850 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:33.576319 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:33.576978 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:33.576986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:33.576992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:33.577105 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:33.577530 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:33.577538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:33.577541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:33.577801 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:33.577951 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.578603 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.578827 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":186,"threads_started":1}
I20260926 07:27:33.580219 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.562369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39319 (local address 127.2.147.193:45739)
0926 07:27:33.563042 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.563077 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.564756 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.565531 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.565542 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.566079 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:33.566682 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.566701 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.568195 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.568200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.569412 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.569422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.570460 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.570466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.570649 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.571361 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:33.571388 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:33.573581 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:33.576452 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:33.576461 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:33.576476 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:33.576861 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:33.577212 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:33.577217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:33.577220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:33.577421 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:33.577928 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:33.577936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:33.578246 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.578523 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.579041 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":260,"threads_started":1}
I20260926 07:27:33.581787 5902 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39319
I20260926 07:27:33.582151 5902 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:33.583086 5902 heartbeater.cc:511] Master 127.2.147.254:39319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:33.585382 5713 ts_manager.cc:194] Registered new tserver with Master: 77b4143adc334b3f85bcbacaf3bb5133 (127.2.147.193:38077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:27:33.587103 5713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:45739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:33.609815 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:33.920481 5908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:33.920742 5908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:33.920925 5908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:33.927947 5908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:33.928050 5908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:33.928265 5908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:33.937985 5908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:33.939383 5908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:33.940683 5908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:33.953758 5913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:33.953778 5916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:33.953853 5914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:33.954398 5908 server_base.cc:1034] running on GCE node
I20260926 07:27:33.955108 5908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:33.956082 5908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:33.957290 5908 hybrid_clock.cc:648] HybridClock initialized: now 1790407653957239 us; error 48 us; skew 500 ppm
I20260926 07:27:33.957621 5908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:33 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407653, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:33.960958 5908 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:33.962574 5908 webserver.cc:460] Webserver started at http://127.2.147.194:32885/ using document root <none> and password file <none>
I20260926 07:27:33.963346 5908 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:33.963488 5908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:33.963794 5908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:33.966730 5908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "0334e146d62c4998b694a2b413d89c2d"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "614295d354bed27c1d8127eabaed6a46"
server_key_iv: "7bdb0e5b8f2ad3f5528173e73e48768e"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.967567 5908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "0334e146d62c4998b694a2b413d89c2d"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "614295d354bed27c1d8127eabaed6a46"
server_key_iv: "7bdb0e5b8f2ad3f5528173e73e48768e"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.972570 5908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 07:27:33.976145 5923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:33.977533 5908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:27:33.977690 5908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "0334e146d62c4998b694a2b413d89c2d"
format_stamp: "Formatted at 2026-09-26 07:27:33 on dist-test-slave-1dwd"
server_key: "614295d354bed27c1d8127eabaed6a46"
server_key_iv: "7bdb0e5b8f2ad3f5528173e73e48768e"
server_key_version: "encryptionkey@0"
I20260926 07:27:33.977872 5908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:33.998970 5908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:34.003513 5908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:34.003844 5908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:34.004781 5908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:34.006064 5908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:34.006146 5908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:34.006294 5908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:34.006349 5908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:34.038702 5908 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:38307
I20260926 07:27:34.038741 6035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:38307 every 8 connection(s)
I20260926 07:27:34.039950 5908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:34.047468 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 5908
I20260926 07:27:34.047634 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:34.048111 2639 external_mini_cluster.cc:1428] Setting key 4b68bff97e94f85637ab0dc090c7406c
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407653, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:34.059634 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.042404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:48465 (local address 127.2.147.254:39319)
0926 07:27:34.042574 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.042578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.043851 (+ 1273us) server_negotiation.cc:407] Connection header received
0926 07:27:34.045521 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.045525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.045588 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.045678 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.047074 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.047963 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.049952 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.050358 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.050489 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.053928 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.053933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.053937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.053966 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.056744 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.057449 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.057456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.057459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.057583 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.057987 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.057994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.057997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.058250 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.058408 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.059050 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.059295 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:27:34.060631 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.042932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39319 (local address 127.2.147.194:48465)
0926 07:27:34.043688 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:34.043729 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:34.045285 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:34.045888 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:34.045895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:34.046365 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:34.046922 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:34.046938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.048136 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.048141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:34.049817 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:34.049827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.050568 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.050594 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:34.050828 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.051600 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:34.051621 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:34.053718 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:34.056887 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:34.056896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:34.056912 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:34.057328 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:34.057692 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:34.057697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:34.057701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:34.057878 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:34.058373 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:34.058382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:34.058717 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 07:27:34.058980 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 07:27:34.059512 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":355,"threads_started":1}
I20260926 07:27:34.062121 6036 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39319
I20260926 07:27:34.062440 6036 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:34.063158 6036 heartbeater.cc:511] Master 127.2.147.254:39319 requested a full tablet report, sending...
I20260926 07:27:34.064687 5713 ts_manager.cc:194] Registered new tserver with Master: 0334e146d62c4998b694a2b413d89c2d (127.2.147.194:38307)
I20260926 07:27:34.065892 5713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:48465
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:34.090081 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:34.403414 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:34.403679 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:34.403832 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:34.410728 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:34.410826 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:34.411021 6041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:34.420743 6041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:34.422055 6041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:34.423374 6041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:34.436112 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:34.436187 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:34.436158 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:34.437013 6041 server_base.cc:1034] running on GCE node
I20260926 07:27:34.437603 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:34.438473 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:34.439685 6041 hybrid_clock.cc:648] HybridClock initialized: now 1790407654439652 us; error 50 us; skew 500 ppm
I20260926 07:27:34.439982 6041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:34.442994 6041 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:34.444478 6041 webserver.cc:460] Webserver started at http://127.2.147.195:46299/ using document root <none> and password file <none>
I20260926 07:27:34.445104 6041 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:34.445268 6041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:34.445591 6041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:34.448417 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "9a8168c6c2e44ecbadd22b34306b5d08"
format_stamp: "Formatted at 2026-09-26 07:27:34 on dist-test-slave-1dwd"
server_key: "0f17cf75402550b28f1ceab256475568"
server_key_iv: "0e2387751e0c6a3d70c0a26877930fbf"
server_key_version: "encryptionkey@0"
I20260926 07:27:34.449141 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8168c6c2e44ecbadd22b34306b5d08"
format_stamp: "Formatted at 2026-09-26 07:27:34 on dist-test-slave-1dwd"
server_key: "0f17cf75402550b28f1ceab256475568"
server_key_iv: "0e2387751e0c6a3d70c0a26877930fbf"
server_key_version: "encryptionkey@0"
I20260926 07:27:34.454027 6041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:27:34.457469 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:34.458678 6041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:27:34.458849 6041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "9a8168c6c2e44ecbadd22b34306b5d08"
format_stamp: "Formatted at 2026-09-26 07:27:34 on dist-test-slave-1dwd"
server_key: "0f17cf75402550b28f1ceab256475568"
server_key_iv: "0e2387751e0c6a3d70c0a26877930fbf"
server_key_version: "encryptionkey@0"
I20260926 07:27:34.459033 6041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:34.474181 6041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:34.477954 6041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:34.478235 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:34.479102 6041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:34.480384 6041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:34.480485 6041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:34.480633 6041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:34.480703 6041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:34.512662 6041 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:34325
I20260926 07:27:34.512707 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:34325 every 8 connection(s)
I20260926 07:27:34.513954 6041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:34.516764 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6041
I20260926 07:27:34.516949 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:34.517421 2639 external_mini_cluster.cc:1428] Setting key 253de55f6a0f7a98a536c0987c6d7f42
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:34.533540 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.516431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:34147 (local address 127.2.147.254:39319)
0926 07:27:34.516650 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.516656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.518015 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 07:27:34.520144 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.520166 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.520268 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.520374 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.521761 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.522481 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.524091 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.524507 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.524653 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.528012 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.528018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.528021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.528052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.530549 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.531299 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.531303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.531306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.531423 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.531854 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.531863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.531865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.532131 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.532277 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.532933 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.533190 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:27:34.534525 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.517015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39319 (local address 127.2.147.195:34147)
0926 07:27:34.517840 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:34.517882 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:34.519864 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:34.520603 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:34.520612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:34.521077 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:34.521608 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:34.521624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.522629 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.522636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:34.523949 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:34.523960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.524739 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.524747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:34.524977 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.525657 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:34.525678 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:34.527830 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:34.530714 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:34.530730 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:34.530743 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:34.531137 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:34.531535 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:34.531541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:34.531544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:34.531735 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:34.532263 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:34.532270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:34.532588 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 07:27:34.532844 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 07:27:34.533384 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260926 07:27:34.535827 6169 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39319
I20260926 07:27:34.536072 6169 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:34.536728 6169 heartbeater.cc:511] Master 127.2.147.254:39319 requested a full tablet report, sending...
I20260926 07:27:34.537955 5713 ts_manager.cc:194] Registered new tserver with Master: 9a8168c6c2e44ecbadd22b34306b5d08 (127.2.147.195:34325)
I20260926 07:27:34.538730 5713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:34147
I20260926 07:27:34.544556 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:34.590204 5902 heartbeater.cc:503] Master 127.2.147.254:39319 was elected leader, sending a full tablet report...
I20260926 07:27:34.590369 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.577241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.2.147.254:39319)
0926 07:27:34.577430 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.577436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.577962 (+ 526us) server_negotiation.cc:407] Connection header received
0926 07:27:34.578113 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.578117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.578177 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.578272 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.579643 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.580318 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.581446 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.581872 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.582040 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.585640 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.585644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.585648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.585676 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.588172 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.588692 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.588696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.588698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.588825 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.589138 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.589141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.589143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.589409 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.589584 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.589811 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.590032 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 07:27:34.590957 5713 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52638
I20260926 07:27:34.592749 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5774
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:34.607785 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.593748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.2.147.193:38077)
0926 07:27:34.594315 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.594322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.594343 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:27:34.594389 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.594395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.594606 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.594768 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.595986 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.596970 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.598131 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.598684 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.598826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.602415 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.602425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.602439 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.602465 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.605159 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.605800 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.605807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.605810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.605922 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.606283 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.606289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.606292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.606610 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.606904 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.607177 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.607435 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
W20260926 07:27:34.608989 5897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46548
W20260926 07:27:34.610054 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5774: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:34.610270 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5774
I20260926 07:27:34.625880 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5908
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:34.640738 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.626727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.2.147.194:38307)
0926 07:27:34.627332 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.627339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.627363 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:27:34.627421 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.627426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.627638 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.627762 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.628850 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.630012 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.631528 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.632043 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.632137 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.635924 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.635933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.635946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.635980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.638432 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.638961 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.638966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.638969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.639083 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.639432 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.639435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.639437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.639749 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.639959 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.640178 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.640360 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":305,"threads_started":1}
W20260926 07:27:34.641995 6031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55564
W20260926 07:27:34.643074 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:34.643416 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5908
I20260926 07:27:34.658833 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6041
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407654, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:34.674976 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:34.659779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.2.147.195:34325)
0926 07:27:34.660417 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:34.660424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:34.660452 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:27:34.660509 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:34.660515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:34.660716 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:34.660890 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:34.662177 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.663267 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.664593 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:34.665160 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:34.665303 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:34.669451 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:34.669460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:34.669475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:34.669506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:34.672155 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.672886 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.672893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.672895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.673016 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:34.673383 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:34.673389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:34.673393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:34.673731 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:34.674052 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:34.674359 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 07:27:34.674632 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":338,"threads_started":1}
W20260926 07:27:34.676388 6164 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37828
W20260926 07:27:34.677544 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6041: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:27:34.677695 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6041
I20260926 07:27:34.693195 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 5681
I20260926 07:27:34.694293 5743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:34.794327 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 5681
2026-09-26T07:27:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3492 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[6186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[6186](info): set up 2 sockets
Sep 26 07:27:34 dist-test-slave-1dwd krb5kdc[6186](info): commencing operation
krb5kdc: starting...
W20260926 07:27:35.863447 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:35 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:36.046108 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40025
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:36.356305 6199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:36.356640 6199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:36.356810 6199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:36.363878 6199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:36.364022 6199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:36.364130 6199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:36.364236 6199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:36.364327 6199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:36.374007 6199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:40025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:36.375355 6199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:36.376639 6199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:36.388155 6204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.388257 6205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.388214 6207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:36.392462 6199 server_base.cc:1034] running on GCE node
I20260926 07:27:36.393203 6199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:36.394526 6199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:36.395748 6199 hybrid_clock.cc:648] HybridClock initialized: now 1790407656395709 us; error 51 us; skew 500 ppm
I20260926 07:27:36.396085 6199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:36 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:36.402321 6199 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:36.403991 6199 webserver.cc:460] Webserver started at http://127.2.147.254:41705/ using document root <none> and password file <none>
I20260926 07:27:36.404670 6199 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:36.404816 6199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:36.405117 6199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:36.408146 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "1b983c43c869493a80f2f88de7176a7f"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "bd7654b5006cd978bb50108979d16f62"
server_key_iv: "95dd541a20f125e0425fd689e6151f7e"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.408947 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "1b983c43c869493a80f2f88de7176a7f"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "bd7654b5006cd978bb50108979d16f62"
server_key_iv: "95dd541a20f125e0425fd689e6151f7e"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.413966 6199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:27:36.417244 6214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:36.418509 6199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:36.418684 6199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "1b983c43c869493a80f2f88de7176a7f"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "bd7654b5006cd978bb50108979d16f62"
server_key_iv: "95dd541a20f125e0425fd689e6151f7e"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.418879 6199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:36.442622 6199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:36.446362 6199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:36.446647 6199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:36.464363 6199 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:40025
I20260926 07:27:36.464401 6265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:40025 every 8 connection(s)
I20260926 07:27:36.465704 6199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:36.469343 6266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:36.473338 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6199
I20260926 07:27:36.473553 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:36.474143 2639 external_mini_cluster.cc:1428] Setting key 975c7e9f2a46f352917a3aa353fb4548
I20260926 07:27:36.481070 6266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f: Bootstrap starting.
I20260926 07:27:36.484664 6266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:36.485977 6266 log.cc:824] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:36.488787 6266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f: No bootstrap required, opened a new log
Sep 26 07:27:36 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:36.498831 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.476130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.2.147.254:40025)
0926 07:27:36.476960 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.476976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.477023 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:27:36.479549 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.479576 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.480046 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.480455 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.481763 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.483821 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.485274 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.486344 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.486547 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.490403 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.490417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.490436 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.490481 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.494485 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.495173 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.495181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.495191 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.495412 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.495741 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.495746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.495749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.496422 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.496713 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.497006 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.497462 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260926 07:27:36.499531 6266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } }
I20260926 07:27:36.499861 6266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:36.499959 6266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b983c43c869493a80f2f88de7176a7f, State: Initialized, Role: FOLLOWER
I20260926 07:27:36.500495 6266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } }
I20260926 07:27:36.500730 6266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:36.500878 6266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:36.501058 6266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:36.503103 6266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } }
I20260926 07:27:36.503623 6266 leader_election.cc:302] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b983c43c869493a80f2f88de7176a7f; no voters:
I20260926 07:27:36.504161 6266 leader_election.cc:288] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:36.504246 6271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:36.504951 6271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [term 1 LEADER]: Becoming Leader. State: Replica: 1b983c43c869493a80f2f88de7176a7f, State: Running, Role: LEADER
I20260926 07:27:36.505537 6271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } }
I20260926 07:27:36.506098 6266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:36.508266 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b983c43c869493a80f2f88de7176a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } } }
I20260926 07:27:36.508723 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:36.509195 6273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b983c43c869493a80f2f88de7176a7f. Latest consensus state: current_term: 1 leader_uuid: "1b983c43c869493a80f2f88de7176a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b983c43c869493a80f2f88de7176a7f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40025 } } }
I20260926 07:27:36.509544 6273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:36.510175 6277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:36.515038 6277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:36.524267 6277 catalog_manager.cc:1347] Generated new cluster ID: 24cacbb5bb3e422cbbf43e0d95c34521
I20260926 07:27:36.524386 6277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:36.543642 6277 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:36.544862 6277 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:36.554811 6277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b983c43c869493a80f2f88de7176a7f: Generated new TSK 0
I20260926 07:27:36.555691 6277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:36.618521 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:36.937178 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:36.937443 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:36.937626 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:36.944741 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:36.944866 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:36.945037 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:36.954931 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:36.956202 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:36.957553 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:36.971045 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.973969 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.973989 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:36.974025 6292 server_base.cc:1034] running on GCE node
I20260926 07:27:36.974699 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:36.975620 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:36.976822 6292 hybrid_clock.cc:648] HybridClock initialized: now 1790407656976778 us; error 35 us; skew 500 ppm
I20260926 07:27:36.977124 6292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:36 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:36.980202 6292 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:36.981627 6292 webserver.cc:460] Webserver started at http://127.2.147.193:44423/ using document root <none> and password file <none>
I20260926 07:27:36.982326 6292 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:36.982470 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:36.982775 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:36.985561 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "30df32336a214996892c3ce5205f4f25"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "f86a77125f79d0c04f325835a2c2e48a"
server_key_iv: "989e292f8e598a2c4cc29b785f0c31b3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.986304 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "30df32336a214996892c3ce5205f4f25"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "f86a77125f79d0c04f325835a2c2e48a"
server_key_iv: "989e292f8e598a2c4cc29b785f0c31b3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.991106 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:27:36.994367 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:36.995604 6292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:36.995779 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "30df32336a214996892c3ce5205f4f25"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-1dwd"
server_key: "f86a77125f79d0c04f325835a2c2e48a"
server_key_iv: "989e292f8e598a2c4cc29b785f0c31b3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.995968 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:37.009969 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:37.014276 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:37.014518 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:37.015409 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:37.016646 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:37.016716 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.016803 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:37.016866 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.048898 6292 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:34471
I20260926 07:27:37.048983 6419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:34471 every 8 connection(s)
I20260926 07:27:37.050192 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:37.058741 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6292
I20260926 07:27:37.058979 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:37.059819 2639 external_mini_cluster.cc:1428] Setting key d2405d387553faea6518721f88e8cea0
Sep 26 07:27:37 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:37.073909 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:37.052669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:34933 (local address 127.2.147.254:40025)
0926 07:27:37.053140 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:37.053151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:37.054128 (+ 977us) server_negotiation.cc:407] Connection header received
0926 07:27:37.056010 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:37.056015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:37.056078 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:37.056175 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:37.057972 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.059228 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.061432 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.062104 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.062230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:37.068107 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:37.068116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:37.068124 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:37.068163 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:37.070819 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:37.071717 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:37.071721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:37.071723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:37.071831 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:37.072279 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:37.072282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:37.072284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:37.072513 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:37.072612 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:37.073504 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 07:27:37.073661 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260926 07:27:37.075505 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:37.053255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40025 (local address 127.2.147.193:34933)
0926 07:27:37.053983 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:37.054024 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:37.055765 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:37.056381 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:37.056389 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:37.056947 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:37.057758 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:37.057783 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.059397 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.059405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:37.061249 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:37.061268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.062362 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.062371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:37.062677 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:37.063525 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:37.063556 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:37.067866 (+ 4310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:37.070958 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:37.070970 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:37.070989 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:37.071615 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:37.071933 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:37.071941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:37.071947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:37.072172 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:37.072635 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:37.072645 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:37.073128 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 07:27:37.073426 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 07:27:37.073977 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":278,"threads_started":1}
I20260926 07:27:37.077028 6420 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40025
I20260926 07:27:37.077330 6420 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:37.078028 6420 heartbeater.cc:511] Master 127.2.147.254:40025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:37.080029 6231 ts_manager.cc:194] Registered new tserver with Master: 30df32336a214996892c3ce5205f4f25 (127.2.147.193:34471)
I20260926 07:27:37.081544 6231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:34933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:37.104624 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:37.421573 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:37.421885 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:37.422066 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:37.429239 6426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:37.429380 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:37.429577 6426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:37.439002 6426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:37.440222 6426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:37.441494 6426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:37.454046 6432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:37.454128 6434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:37.454114 6431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:37.454847 6426 server_base.cc:1034] running on GCE node
I20260926 07:27:37.455454 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:37.456221 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:37.457408 6426 hybrid_clock.cc:648] HybridClock initialized: now 1790407657457359 us; error 39 us; skew 500 ppm
I20260926 07:27:37.457664 6426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:37 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:37.460625 6426 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:37.461927 6426 webserver.cc:460] Webserver started at http://127.2.147.194:40043/ using document root <none> and password file <none>
I20260926 07:27:37.462527 6426 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:37.462680 6426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:37.462980 6426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:37.465852 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "1ef18628428b4a2da23fe10baf79a0a1"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "b69c58868e112d0b624e1b74d67522fe"
server_key_iv: "94c476d7fda502e789375550c8c9f8bf"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.466589 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "1ef18628428b4a2da23fe10baf79a0a1"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "b69c58868e112d0b624e1b74d67522fe"
server_key_iv: "94c476d7fda502e789375550c8c9f8bf"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.471467 6426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 07:27:37.474834 6441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.476104 6426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:37.476269 6426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "1ef18628428b4a2da23fe10baf79a0a1"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "b69c58868e112d0b624e1b74d67522fe"
server_key_iv: "94c476d7fda502e789375550c8c9f8bf"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.476449 6426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:37.488654 6426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:37.492204 6426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:37.492487 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:37.493409 6426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:37.494601 6426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:37.494699 6426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.494843 6426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:37.494918 6426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.526826 6426 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:45883
I20260926 07:27:37.526870 6553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:45883 every 8 connection(s)
I20260926 07:27:37.528127 6426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:37.533876 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6426
I20260926 07:27:37.534075 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:37.534480 2639 external_mini_cluster.cc:1428] Setting key 9cb672aca43b07214864315efc5f08d4
Sep 26 07:27:37 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:37.549118 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:37.530553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:57119 (local address 127.2.147.254:40025)
0926 07:27:37.530710 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:37.530714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:37.531779 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 07:27:37.533515 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:37.533520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:37.533632 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:37.533741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:37.535486 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.536328 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.538799 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.539261 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.539371 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:37.543484 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:37.543494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:37.543501 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:37.543543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:37.546056 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:37.546952 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:37.546956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:37.546958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:37.547068 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:37.547549 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:37.547552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:37.547554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:37.547789 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:37.547898 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:37.548721 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 07:27:37.548872 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:37.550696 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:37.530925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40025 (local address 127.2.147.194:57119)
0926 07:27:37.531625 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:37.531673 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:37.533250 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:37.533970 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:37.533978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:37.534583 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:37.535318 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:37.535338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.536489 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.536494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:37.538657 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:37.538676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:37.539488 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:37.539497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:37.539826 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:37.540529 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:37.540561 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:37.543278 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:37.546197 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:37.546208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:37.546226 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:37.546837 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:37.547173 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:37.547180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:37.547186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:37.547440 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:37.547917 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:37.547927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:37.548357 (+ 430us) client_negotiation.cc:769] Sending connection context
0926 07:27:37.548647 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 07:27:37.549204 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260926 07:27:37.552158 6554 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40025
I20260926 07:27:37.552459 6554 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:37.553177 6554 heartbeater.cc:511] Master 127.2.147.254:40025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:27:37.554744 6231 ts_manager.cc:194] Registered new tserver with Master: 1ef18628428b4a2da23fe10baf79a0a1 (127.2.147.194:45883)
I20260926 07:27:37.555678 6231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:57119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:37.577706 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:37.889014 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:37.889279 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:37.889467 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:37.896654 6559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:37.896780 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:37.896953 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:37.906411 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:37.907835 6559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:37.909129 6559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:37.922242 6565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:37.922358 6567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:37.922327 6564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:37.923467 6559 server_base.cc:1034] running on GCE node
I20260926 07:27:37.924034 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:37.924952 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:37.926174 6559 hybrid_clock.cc:648] HybridClock initialized: now 1790407657926125 us; error 65 us; skew 500 ppm
I20260926 07:27:37.926479 6559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:37 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:37.929425 6559 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:37.930815 6559 webserver.cc:460] Webserver started at http://127.2.147.195:37941/ using document root <none> and password file <none>
I20260926 07:27:37.931535 6559 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:37.931675 6559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:37.931973 6559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:37.934801 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "b6cb458070884e33b38a863eb49f2380"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "dc50cd208e9673d00e78adccd1e05a90"
server_key_iv: "21d75c63fdc0bb521e45efddc6262897"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.935600 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "b6cb458070884e33b38a863eb49f2380"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "dc50cd208e9673d00e78adccd1e05a90"
server_key_iv: "21d75c63fdc0bb521e45efddc6262897"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.940348 6559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:27:37.943665 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.944916 6559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:37.945083 6559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "b6cb458070884e33b38a863eb49f2380"
format_stamp: "Formatted at 2026-09-26 07:27:37 on dist-test-slave-1dwd"
server_key: "dc50cd208e9673d00e78adccd1e05a90"
server_key_iv: "21d75c63fdc0bb521e45efddc6262897"
server_key_version: "encryptionkey@0"
I20260926 07:27:37.945284 6559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:37.961609 6559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:37.965440 6559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:37.965726 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:37.966636 6559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:37.967976 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:37.968078 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:37.968231 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:37.968321 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:38.000655 6559 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:33801
I20260926 07:27:38.000722 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:33801 every 8 connection(s)
I20260926 07:27:38.002025 6559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:38.003855 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6559
I20260926 07:27:38.004055 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:38.004525 2639 external_mini_cluster.cc:1428] Setting key f67ae70aa4bc59fa245287e6fbca70ba
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:38.021495 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.004597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:37549 (local address 127.2.147.254:40025)
0926 07:27:38.004802 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.004808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.006290 (+ 1482us) server_negotiation.cc:407] Connection header received
0926 07:27:38.008508 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.008512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.008577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.008656 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.010068 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.010794 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.012298 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.012654 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.012743 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.015919 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.015924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.015927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.015956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.018736 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.019594 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.019600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.019603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.019767 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.020136 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.020139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.020141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.020388 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.020494 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.021087 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.021242 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:27:38.022357 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.005242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40025 (local address 127.2.147.195:37549)
0926 07:27:38.006122 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:38.006182 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:38.008272 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:38.008860 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:38.008868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:38.009381 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:38.009932 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:38.009949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.010915 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.010919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:38.012185 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:38.012196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.012828 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.012832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:38.013011 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.013702 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:38.013727 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:38.015744 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:38.018856 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:38.018862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:38.018873 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:38.019424 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:38.019864 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:38.019867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:38.019870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:38.020039 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:38.020488 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:38.020493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:38.020774 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 07:27:38.020983 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:27:38.021304 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":340,"threads_started":1}
I20260926 07:27:38.023660 6687 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40025
I20260926 07:27:38.023957 6687 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:38.024621 6687 heartbeater.cc:511] Master 127.2.147.254:40025 requested a full tablet report, sending...
I20260926 07:27:38.025902 6231 ts_manager.cc:194] Registered new tserver with Master: b6cb458070884e33b38a863eb49f2380 (127.2.147.195:33801)
I20260926 07:27:38.026666 6231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:37549
I20260926 07:27:38.031746 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:38.077613 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.063090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.2.147.193:34471)
0926 07:27:38.063742 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.063754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.063802 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:27:38.063871 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.063877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.064073 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.064239 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.065564 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.066522 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.067766 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.068316 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.068457 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.072278 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.072287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.072299 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.072325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.075027 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.075693 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.075700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.075702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.075812 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.076180 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.076189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.076191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.076482 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.076803 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.077038 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.077274 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":337,"threads_started":1}
I20260926 07:27:38.085002 6420 heartbeater.cc:503] Master 127.2.147.254:40025 was elected leader, sending a full tablet report...
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:38.094808 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.081599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.2.147.254:40025)
0926 07:27:38.081808 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.081814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.081970 (+ 156us) server_negotiation.cc:407] Connection header received
0926 07:27:38.082158 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.082163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.082256 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.082359 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.083555 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.084275 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.085699 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.086108 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.086244 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.089962 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.089967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.089970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.090004 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.092309 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.092895 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.092907 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.092910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.093034 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.093404 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.093411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.093414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.093714 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.093959 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.094281 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.094492 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:27:38.095347 6231 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39180
I20260926 07:27:38.096276 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6292
I20260926 07:27:38.108304 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.097058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.2.147.193:34471)
0926 07:27:38.097224 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.097228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.097564 (+ 336us) server_negotiation.cc:407] Connection header received
0926 07:27:38.097674 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.097678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.097740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.097818 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.098878 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.099620 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.100964 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.101358 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.101445 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.103588 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.103592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.103595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.103623 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.106180 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.106745 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.106748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.106750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.106862 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.107335 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.107338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.107340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.107559 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.107724 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.107939 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.108082 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:38.108816 6415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:38.191517 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6292
I20260926 07:27:38.209789 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6426
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:38.224959 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.210640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.2.147.194:45883)
0926 07:27:38.211240 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.211249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.211284 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:27:38.211344 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.211350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.211568 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.211690 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.212810 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.213797 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.214997 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.215607 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.215707 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.219748 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.219757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.219794 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.219823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.222425 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.223087 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.223093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.223095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.223309 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.223647 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.223650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.223652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.223942 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.224135 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.224416 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.224613 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260926 07:27:38.226104 6549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:38.306000 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6426
I20260926 07:27:38.322892 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6559
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:38.338217 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:38.323916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.2.147.195:33801)
0926 07:27:38.324461 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:38.324468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:38.324495 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:27:38.324553 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:38.324559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:38.324765 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:38.324930 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:38.326155 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.327154 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.328463 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:38.329011 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:38.329111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:38.333137 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:38.333148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:38.333161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:38.333191 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:38.335844 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.336502 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.336509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.336511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.336620 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:38.336912 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:38.336915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:38.336917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:38.337205 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:38.337409 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:38.337653 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 07:27:38.337837 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":281,"threads_started":1}
I20260926 07:27:38.339459 6682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:38.421942 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6559
I20260926 07:27:38.437595 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6199
I20260926 07:27:38.438743 6261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:38.540951 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6199
2026-09-26T07:27:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3745 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6706](info): set up 2 sockets
Sep 26 07:27:38 dist-test-slave-1dwd krb5kdc[6706](info): commencing operation
krb5kdc: starting...
W20260926 07:27:39.619581 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:39 dist-test-slave-1dwd krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:39.796469 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39775
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:39775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:40.100549 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:40.100819 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:40.101006 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:40.108054 6719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:40.108145 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:40.108239 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:40.108349 6719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:40.108436 6719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:40.118032 6719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:39775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:40.119374 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:40.120690 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:40.132215 6727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.132236 6725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.132270 6724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.132958 6719 server_base.cc:1034] running on GCE node
I20260926 07:27:40.133760 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:40.135079 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:40.136305 6719 hybrid_clock.cc:648] HybridClock initialized: now 1790407660136269 us; error 54 us; skew 500 ppm
I20260926 07:27:40.136647 6719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:40 dist-test-slave-1dwd krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:40.139699 6719 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:40.140815 6719 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:40.140956 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:40.141278 6719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:40.144212 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "dcc7a38eeb694eb091b84d219b35ec86"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "7dfbf4c51edce2ba8c361a34adc61e39"
server_key_iv: "2747e7994e34c7ac04267b837a511852"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.144989 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "dcc7a38eeb694eb091b84d219b35ec86"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "7dfbf4c51edce2ba8c361a34adc61e39"
server_key_iv: "2747e7994e34c7ac04267b837a511852"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.149817 6719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:40.153023 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.154227 6719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:27:40.154412 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "dcc7a38eeb694eb091b84d219b35ec86"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "7dfbf4c51edce2ba8c361a34adc61e39"
server_key_iv: "2747e7994e34c7ac04267b837a511852"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.154614 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:40.171756 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:40.175828 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:40.176074 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:40.193414 6719 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:39775
I20260926 07:27:40.193429 6784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:39775 every 8 connection(s)
I20260926 07:27:40.194250 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:40.197643 6785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:40.202025 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6719
I20260926 07:27:40.202247 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:40.202885 2639 external_mini_cluster.cc:1428] Setting key 57d1deef34f6c890a61c301e87ec3413
I20260926 07:27:40.207809 6785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: Bootstrap starting.
I20260926 07:27:40.210530 6785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:40.211534 6785 log.cc:824] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:40.214363 6785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: No bootstrap required, opened a new log
Sep 26 07:27:40 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:40.223344 6785 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } }
I20260926 07:27:40.223692 6785 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:40.223791 6785 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc7a38eeb694eb091b84d219b35ec86, State: Initialized, Role: FOLLOWER
I20260926 07:27:40.224395 6785 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } }
I20260926 07:27:40.224612 6785 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:40.224759 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.204801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.2.147.254:39775)
0926 07:27:40.205617 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:40.205628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:40.205669 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:27:40.207336 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:40.207356 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:40.207688 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:40.208028 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:40.209499 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.210824 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.212286 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.213149 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.213318 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.217569 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:40.217577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:40.217593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:40.217625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:40.220601 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.221361 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.221366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.221372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.221525 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.221939 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.221942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.221944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.222406 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:40.222621 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:40.223058 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 07:27:40.223470 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":381,"threads_started":1}
I20260926 07:27:40.224783 6785 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:40.224973 6785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:40.227177 6785 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } }
I20260926 07:27:40.227741 6785 leader_election.cc:302] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc7a38eeb694eb091b84d219b35ec86; no voters:
I20260926 07:27:40.228272 6785 leader_election.cc:288] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:40.228401 6790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:40.229075 6790 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [term 1 LEADER]: Becoming Leader. State: Replica: dcc7a38eeb694eb091b84d219b35ec86, State: Running, Role: LEADER
I20260926 07:27:40.229580 6790 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } }
I20260926 07:27:40.230314 6785 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:40.232394 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc7a38eeb694eb091b84d219b35ec86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } } }
I20260926 07:27:40.232419 6792 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc7a38eeb694eb091b84d219b35ec86. Latest consensus state: current_term: 1 leader_uuid: "dcc7a38eeb694eb091b84d219b35ec86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc7a38eeb694eb091b84d219b35ec86" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39775 } } }
I20260926 07:27:40.232772 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:40.232784 6792 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:40.233624 6795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:40.239522 6795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:40.248351 6795 catalog_manager.cc:1347] Generated new cluster ID: 72b8dbecc6314e0299fb91605a25649e
I20260926 07:27:40.248476 6795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:40.258641 6795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:40.259979 6795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:40.265691 6795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: Generated new TSK 0
I20260926 07:27:40.266455 6795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:40.318761 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:40.651968 6811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:40.652210 6811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:40.652299 6811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:40.659631 6811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:40.659720 6811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:40.659915 6811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:40.669094 6811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:40.670257 6811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:40.671947 6811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:40.684195 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.684283 6819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.684337 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.685269 6811 server_base.cc:1034] running on GCE node
I20260926 07:27:40.685807 6811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:40.686609 6811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:40.687841 6811 hybrid_clock.cc:648] HybridClock initialized: now 1790407660687796 us; error 47 us; skew 500 ppm
I20260926 07:27:40.688129 6811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:40 dist-test-slave-1dwd krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:40.691128 6811 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:40.692205 6811 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:40.692333 6811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:40.692626 6811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:40.695428 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "3710657d71814e9a957d8991c04a0f53"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "343240a80bc3001fb4f2a1d60564d7cf"
server_key_iv: "50cab320b51c83900894a67c1a46652c"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.696213 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "3710657d71814e9a957d8991c04a0f53"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "343240a80bc3001fb4f2a1d60564d7cf"
server_key_iv: "50cab320b51c83900894a67c1a46652c"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.700752 6811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:40.703861 6825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.705036 6811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:40.705224 6811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "3710657d71814e9a957d8991c04a0f53"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-1dwd"
server_key: "343240a80bc3001fb4f2a1d60564d7cf"
server_key_iv: "50cab320b51c83900894a67c1a46652c"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.705430 6811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:40.724375 6811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:40.728533 6811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:40.728788 6811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:40.729650 6811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:40.730984 6811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:40.731098 6811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.731252 6811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:40.731354 6811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 07:27:40.762794 6811 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38325
I20260926 07:27:40.762835 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38325 every 8 connection(s)
I20260926 07:27:40.763692 6811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:40.769687 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6811
I20260926 07:27:40.769888 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:40.770509 2639 external_mini_cluster.cc:1428] Setting key 1e186a8221e92a359ed88bfc2f4efde5
Sep 26 07:27:40 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:40.782717 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.765919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:35767 (local address 127.2.147.254:39775)
0926 07:27:40.766287 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:40.766291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:40.767275 (+ 984us) server_negotiation.cc:407] Connection header received
0926 07:27:40.768943 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:40.768948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:40.769030 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:40.769142 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:40.770706 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.771588 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.773214 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.773604 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.773735 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.777027 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:40.777036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:40.777054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:40.777094 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:40.779796 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.780664 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.780668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.780670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.780768 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.781185 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.781187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.781189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.781420 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:40.781570 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:40.782342 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 07:27:40.782479 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":203,"threads_started":1}
I20260926 07:27:40.784215 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.766449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39775 (local address 127.2.147.193:35767)
0926 07:27:40.767104 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:40.767144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:40.768692 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:40.769395 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:40.769404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:40.769972 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:40.770518 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:40.770536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.771731 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.771738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:40.773034 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:40.773059 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.773813 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.773819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:40.774039 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.774726 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:40.774746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:40.776821 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:40.779923 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:40.779933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:40.779951 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:40.780567 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:40.780867 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:40.780874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:40.780881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:40.781086 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:40.781558 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:40.781568 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:40.781998 (+ 430us) client_negotiation.cc:769] Sending connection context
0926 07:27:40.782281 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 07:27:40.782805 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":267,"threads_started":1}
I20260926 07:27:40.785609 6938 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39775
I20260926 07:27:40.785957 6938 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:40.786795 6938 heartbeater.cc:511] Master 127.2.147.254:39775 requested a full tablet report, sending...
I20260926 07:27:40.788844 6750 ts_manager.cc:194] Registered new tserver with Master: 3710657d71814e9a957d8991c04a0f53 (127.2.147.193:38325)
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:40.790395 6750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:35767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:40.813565 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:41.121577 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:41.121836 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:41.121958 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:41.128974 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:41.129072 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:41.129241 6944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:41.138935 6944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:41.140218 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:41.141552 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:41.153795 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.153852 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.153898 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.154490 6944 server_base.cc:1034] running on GCE node
I20260926 07:27:41.155103 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:41.155997 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:41.157183 6944 hybrid_clock.cc:648] HybridClock initialized: now 1790407661157160 us; error 33 us; skew 500 ppm
I20260926 07:27:41.157485 6944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:41 dist-test-slave-1dwd krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.160555 6944 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:41.161554 6944 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.161638 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.161875 6944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:41.164685 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "59404dce922d4260a5eea7022e6fcebc"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "d37cfbfadff1281d555e59bc59d6d19e"
server_key_iv: "9a736b762e14393f9c34122dd75e598b"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.165462 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "59404dce922d4260a5eea7022e6fcebc"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "d37cfbfadff1281d555e59bc59d6d19e"
server_key_iv: "9a736b762e14393f9c34122dd75e598b"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.170120 6944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 07:27:41.173311 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.174464 6944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:41.174624 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "59404dce922d4260a5eea7022e6fcebc"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "d37cfbfadff1281d555e59bc59d6d19e"
server_key_iv: "9a736b762e14393f9c34122dd75e598b"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.174846 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.226192 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.230540 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.230831 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.231935 6944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:41.233264 6944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:41.233397 6944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.233525 6944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:41.233611 6944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.266533 6944 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37643
I20260926 07:27:41.266619 7070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37643 every 8 connection(s)
I20260926 07:27:41.267493 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:41.269506 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 6944
I20260926 07:27:41.269688 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:41.270259 2639 external_mini_cluster.cc:1428] Setting key f956d1d0f5db02377f74739673fcfbb4
Sep 26 07:27:41 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:41.290647 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.269973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:42645 (local address 127.2.147.254:39775)
0926 07:27:41.270188 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.270194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.271572 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 07:27:41.274079 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.274083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.274142 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.274225 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.276370 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.277065 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.279353 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.280078 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.280218 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.283719 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.283729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.283736 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.283777 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.287453 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.288182 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.288190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.288196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.288375 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.288793 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.288799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.288804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.289177 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.289336 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.289998 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.290226 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:27:41.291409 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.270571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39775 (local address 127.2.147.194:42645)
0926 07:27:41.271431 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.271472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.273791 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.274517 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.274531 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.275343 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:41.276187 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.276213 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.277218 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.277225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.279186 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.279201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.280282 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.280287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.280542 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.281227 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:41.281249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:41.283479 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:41.287586 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.287593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.287606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.288013 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.288488 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.288491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.288493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.288663 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.289296 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:41.289302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:41.289611 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.289834 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.290224 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":370,"threads_started":1}
I20260926 07:27:41.292769 7071 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39775
I20260926 07:27:41.293093 7071 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:41.293803 7071 heartbeater.cc:511] Master 127.2.147.254:39775 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
I20260926 07:27:41.295336 6750 ts_manager.cc:194] Registered new tserver with Master: 59404dce922d4260a5eea7022e6fcebc (127.2.147.194:37643)
I20260926 07:27:41.296105 6750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:42645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:41.316654 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:41.629645 7076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:41.629935 7076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:41.630029 7076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:41.637367 7076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:41.637468 7076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:41.637586 7076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:41.647320 7076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:41.648607 7076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:41.649833 7076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:41.663466 7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.663548 7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.663522 7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.664469 7076 server_base.cc:1034] running on GCE node
I20260926 07:27:41.665129 7076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:41.666074 7076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:41.667315 7076 hybrid_clock.cc:648] HybridClock initialized: now 1790407661667264 us; error 47 us; skew 500 ppm
I20260926 07:27:41.667629 7076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:41 dist-test-slave-1dwd krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.675753 7076 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:41.677013 7076 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.677146 7076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.677495 7076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:41.680666 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "0db74dc71b3941b29f2fe99ecf5b3674"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "662235b1f6fa694db835ab17c4f91223"
server_key_iv: "319e3f3c0278c29d5cce9fea46aa1ec8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.681538 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "0db74dc71b3941b29f2fe99ecf5b3674"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "662235b1f6fa694db835ab17c4f91223"
server_key_iv: "319e3f3c0278c29d5cce9fea46aa1ec8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.686429 7076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:27:41.689844 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.691143 7076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:41.691339 7076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "0db74dc71b3941b29f2fe99ecf5b3674"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-1dwd"
server_key: "662235b1f6fa694db835ab17c4f91223"
server_key_iv: "319e3f3c0278c29d5cce9fea46aa1ec8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.691557 7076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.707614 7076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.711416 7076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.711674 7076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.712539 7076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:41.713765 7076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:41.713892 7076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.714020 7076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:41.714105 7076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.746131 7076 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:40297
I20260926 07:27:41.746186 7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:40297 every 8 connection(s)
I20260926 07:27:41.746986 7076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:41.755544 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7076
I20260926 07:27:41.755734 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:41.756150 2639 external_mini_cluster.cc:1428] Setting key 4c081f9bdcd04367921f813deed33809
Sep 26 07:27:41 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:41.766024 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.749424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43171 (local address 127.2.147.254:39775)
0926 07:27:41.749597 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.749601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.750706 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 07:27:41.752427 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.752431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.752493 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.752579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.753974 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.754690 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.756330 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.756881 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.756978 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.760638 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.760643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.760646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.760673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.763140 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.763879 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.763884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.763888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.764011 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.764421 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.764425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.764429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.764683 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.764854 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.765475 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.765678 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:27:41.766889 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.749901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39775 (local address 127.2.147.195:43171)
0926 07:27:41.750538 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.750579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.752188 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.752773 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.752781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.753270 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:41.753815 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.753835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.754808 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.754812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.756173 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.756184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.757084 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.757092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.757358 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.758125 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:41.758160 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:41.760474 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:41.763327 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.763335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.763348 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.763755 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.764118 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.764124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.764127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.764304 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.764831 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:41.764838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:41.765140 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.765389 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.765858 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
I20260926 07:27:41.768119 7203 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39775
I20260926 07:27:41.768428 7203 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:41.769057 7203 heartbeater.cc:511] Master 127.2.147.254:39775 requested a full tablet report, sending...
I20260926 07:27:41.770493 6750 ts_manager.cc:194] Registered new tserver with Master: 0db74dc71b3941b29f2fe99ecf5b3674 (127.2.147.195:40297)
I20260926 07:27:41.771512 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:41.771615 6750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:43171
I20260926 07:27:41.788849 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.776809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.2.147.254:39775)
0926 07:27:41.776964 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.776968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.777455 (+ 487us) server_negotiation.cc:407] Connection header received
0926 07:27:41.777659 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.777665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.777733 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.777833 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.779304 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.779976 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.781293 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.781729 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.781862 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.784014 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.784020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.784025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.784065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.786301 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.786924 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.786931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.786934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.787089 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.787477 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.787484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.787488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.787765 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.788028 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.788347 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.788546 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:27:41.793164 6938 heartbeater.cc:503] Master 127.2.147.254:39775 was elected leader, sending a full tablet report...
I20260926 07:27:41.792538 6750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:41.799593 6750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:41.823055 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.810230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:40297 (local address 127.0.0.1:45692)
0926 07:27:41.810920 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.810969 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.811159 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.813144 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.813154 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.813194 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.813660 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.813674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.816257 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.816265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.819340 (+ 3075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.819359 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.822117 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.822126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.822367 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.822424 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.822545 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.822716 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":1471,"threads_started":1}
I20260926 07:27:41.824040 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.810454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.2.147.195:40297)
0926 07:27:41.812083 (+ 1629us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.812095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.812134 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:41.812248 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.812259 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.812537 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.812756 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.813917 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.816055 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.819580 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.821505 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.821659 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.821956 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.823524 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.823679 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1453,"thread_start_us":1318,"threads_started":1}
I20260926 07:27:41.828847 7138 tablet_service.cc:1467] Processing CreateTablet for tablet e04b90e259224eb58707f0ba494df26c (DEFAULT_TABLE table=test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.830262 7138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04b90e259224eb58707f0ba494df26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.830720 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.810686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38325 (local address 127.0.0.1:38016)
0926 07:27:41.811338 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.811361 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.811501 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.823609 (+ 12108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.823614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.823639 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.823992 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.824003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.826224 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.826228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.828319 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.828332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.829835 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.829841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.830047 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.830082 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.830190 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.830320 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":611,"threads_started":1}
I20260926 07:27:41.831382 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.811976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37643 (local address 127.0.0.1:59148)
0926 07:27:41.823291 (+ 11315us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.823306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.823384 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.825449 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.825453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.825475 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.825677 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.825687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.827027 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.827032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.828895 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.828909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.830656 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.830663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.830833 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.830863 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.830937 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.831088 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11241,"thread_start_us":12434,"threads_started":1}
I20260926 07:27:41.830695 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.811069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.2.147.193:38325)
0926 07:27:41.822615 (+ 11546us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.822622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.822647 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:27:41.822743 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.822750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.822972 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.824369 (+ 1397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.824598 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.826082 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.828474 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.829640 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.829805 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.830026 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.830241 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.830388 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11334,"thread_start_us":12254,"threads_started":1}
I20260926 07:27:41.832314 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.812274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.2.147.194:37643)
0926 07:27:41.824954 (+ 12680us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.824962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.824987 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:27:41.825068 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.825074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.825269 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.825438 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.825823 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.826884 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.829065 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.830472 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.831618 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.831789 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.831850 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.831991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12514,"thread_start_us":12367,"threads_started":1}
I20260926 07:27:41.835078 6873 tablet_service.cc:1467] Processing CreateTablet for tablet e04b90e259224eb58707f0ba494df26c (DEFAULT_TABLE table=test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.835642 7006 tablet_service.cc:1467] Processing CreateTablet for tablet e04b90e259224eb58707f0ba494df26c (DEFAULT_TABLE table=test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.836453 6873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04b90e259224eb58707f0ba494df26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.836879 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04b90e259224eb58707f0ba494df26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.842250 7221 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Bootstrap starting.
I20260926 07:27:41.845170 7221 tablet_bootstrap.cc:654] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.846453 7221 log.cc:824] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.850831 7224 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Bootstrap starting.
I20260926 07:27:41.854635 7221 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: No bootstrap required, opened a new log
I20260926 07:27:41.854918 7221 ts_tablet_manager.cc:1397] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260926 07:27:41.855479 7223 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Bootstrap starting.
I20260926 07:27:41.856180 7224 tablet_bootstrap.cc:654] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.857465 7224 log.cc:824] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.858510 7223 tablet_bootstrap.cc:654] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.859552 7223 log.cc:824] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.860463 7224 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: No bootstrap required, opened a new log
I20260926 07:27:41.860775 7224 ts_tablet_manager.cc:1397] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 07:27:41.865226 7223 tablet_bootstrap.cc:492] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: No bootstrap required, opened a new log
I20260926 07:27:41.865494 7223 ts_tablet_manager.cc:1397] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260926 07:27:41.865518 7221 raft_consensus.cc:348] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.865958 7221 raft_consensus.cc:374] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.866060 7221 raft_consensus.cc:729] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db74dc71b3941b29f2fe99ecf5b3674, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.866663 7221 consensus_queue.cc:260] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.867748 7203 heartbeater.cc:503] Master 127.2.147.254:39775 was elected leader, sending a full tablet report...
I20260926 07:27:41.869581 7221 ts_tablet_manager.cc:1428] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 07:27:41.870540 7224 raft_consensus.cc:348] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.870878 7224 raft_consensus.cc:374] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.870988 7224 raft_consensus.cc:729] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59404dce922d4260a5eea7022e6fcebc, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.871610 7224 consensus_queue.cc:260] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.872802 7071 heartbeater.cc:503] Master 127.2.147.254:39775 was elected leader, sending a full tablet report...
I20260926 07:27:41.874778 7224 ts_tablet_manager.cc:1428] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260926 07:27:41.875320 7223 raft_consensus.cc:348] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.875636 7223 raft_consensus.cc:374] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.875742 7223 raft_consensus.cc:729] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3710657d71814e9a957d8991c04a0f53, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.876344 7223 consensus_queue.cc:260] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:41.877765 7223 ts_tablet_manager.cc:1428] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260926 07:27:41.998558 7204 tablet.cc:2369] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:42.015766 6939 tablet.cc:2369] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:42.019366 7072 tablet.cc:2369] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:42.022050 7229 raft_consensus.cc:482] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:42.022292 7229 raft_consensus.cc:504] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:42.023869 7229 leader_election.cc:288] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db74dc71b3941b29f2fe99ecf5b3674 (127.2.147.195:40297), 59404dce922d4260a5eea7022e6fcebc (127.2.147.194:37643)
I20260926 07:27:42.030822 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.024381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50275 (local address 127.2.147.195:40297)
0926 07:27:42.024622 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.024629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.024955 (+ 326us) server_negotiation.cc:407] Connection header received
0926 07:27:42.025170 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.025176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.025250 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.025339 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.026021 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.026960 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.028870 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.029813 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.029946 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.030058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.030424 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.030573 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:27:42.030905 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.024196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:40297 (local address 127.2.147.193:50275)
0926 07:27:42.024844 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.024859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.025026 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.025405 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.025413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.025451 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.025857 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.025875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.027082 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.027088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.028725 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.028736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.030044 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.030050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.030253 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.030300 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.030386 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.030549 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":2,"thread_start_us":362,"threads_started":1}
I20260926 07:27:42.031246 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.025102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50261 (local address 127.2.147.194:37643)
0926 07:27:42.025320 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.025327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.025728 (+ 401us) server_negotiation.cc:407] Connection header received
0926 07:27:42.025868 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.025873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.025955 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.026042 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.026609 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.027488 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.029323 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.030238 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.030383 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.030445 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.030830 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.030971 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:27:42.031387 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.024985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37643 (local address 127.2.147.193:50261)
0926 07:27:42.025619 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.025632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.025737 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.026080 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.026087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.026113 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.026456 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.026466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.027631 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.027634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.029172 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.029188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.030472 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.030478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.030652 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.030685 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.030783 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.030915 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":43,"thread_start_us":349,"threads_started":1}
I20260926 07:27:42.031929 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04b90e259224eb58707f0ba494df26c" candidate_uuid: "3710657d71814e9a957d8991c04a0f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" is_pre_election: true
I20260926 07:27:42.032397 7158 raft_consensus.cc:2454] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710657d71814e9a957d8991c04a0f53 in term 0.
I20260926 07:27:42.032348 7026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04b90e259224eb58707f0ba494df26c" candidate_uuid: "3710657d71814e9a957d8991c04a0f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59404dce922d4260a5eea7022e6fcebc" is_pre_election: true
I20260926 07:27:42.032814 7026 raft_consensus.cc:2454] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3710657d71814e9a957d8991c04a0f53 in term 0.
I20260926 07:27:42.033011 6828 leader_election.cc:302] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db74dc71b3941b29f2fe99ecf5b3674, 3710657d71814e9a957d8991c04a0f53; no voters:
I20260926 07:27:42.033412 7229 raft_consensus.cc:2781] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:42.033553 7229 raft_consensus.cc:482] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:42.033667 7229 raft_consensus.cc:3037] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.035876 7229 raft_consensus.cc:504] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:42.036463 7229 leader_election.cc:288] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [CANDIDATE]: Term 1 election: Requested vote from peers 0db74dc71b3941b29f2fe99ecf5b3674 (127.2.147.195:40297), 59404dce922d4260a5eea7022e6fcebc (127.2.147.194:37643)
I20260926 07:27:42.036777 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04b90e259224eb58707f0ba494df26c" candidate_uuid: "3710657d71814e9a957d8991c04a0f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db74dc71b3941b29f2fe99ecf5b3674"
I20260926 07:27:42.036926 7026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04b90e259224eb58707f0ba494df26c" candidate_uuid: "3710657d71814e9a957d8991c04a0f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59404dce922d4260a5eea7022e6fcebc"
I20260926 07:27:42.036967 7158 raft_consensus.cc:3037] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.037114 7026 raft_consensus.cc:3037] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.039052 7158 raft_consensus.cc:2454] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710657d71814e9a957d8991c04a0f53 in term 1.
I20260926 07:27:42.039238 7026 raft_consensus.cc:2454] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3710657d71814e9a957d8991c04a0f53 in term 1.
I20260926 07:27:42.039434 6828 leader_election.cc:302] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db74dc71b3941b29f2fe99ecf5b3674, 3710657d71814e9a957d8991c04a0f53; no voters:
I20260926 07:27:42.039685 7229 raft_consensus.cc:2781] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:42.040093 7229 raft_consensus.cc:686] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 1 LEADER]: Becoming Leader. State: Replica: 3710657d71814e9a957d8991c04a0f53, State: Running, Role: LEADER
I20260926 07:27:42.040557 7229 consensus_queue.cc:237] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } }
I20260926 07:27:42.045068 6749 catalog_manager.cc:5495] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3710657d71814e9a957d8991c04a0f53 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "3710657d71814e9a957d8991c04a0f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3710657d71814e9a957d8991c04a0f53" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38325 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:42.069489 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.063761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.2.147.193:38325)
0926 07:27:42.063943 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.063949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.064045 (+ 96us) server_negotiation.cc:407] Connection header received
0926 07:27:42.064278 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.064282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.064358 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.064499 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:42.064833 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.065502 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.066975 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.067783 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.067914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.068482 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:42.068954 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:42.069110 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.069163 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.069234 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:27:42.078960 7026 raft_consensus.cc:1261] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 1 FOLLOWER]: Refusing update from remote peer 3710657d71814e9a957d8991c04a0f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:42.079042 7158 raft_consensus.cc:1261] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 1 FOLLOWER]: Refusing update from remote peer 3710657d71814e9a957d8991c04a0f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:42.079918 7232 consensus_queue.cc:1035] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59404dce922d4260a5eea7022e6fcebc" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:42.080256 7229 consensus_queue.cc:1035] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db74dc71b3941b29f2fe99ecf5b3674" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:42.090724 7236 mvcc.cc:205] Tried to move back new op lower bound from 7333509783866191872 to 7333509783721611264. Current Snapshot: MvccSnapshot[applied={T|T < 7333509783866191872}]
I20260926 07:27:42.095479 7237 mvcc.cc:205] Tried to move back new op lower bound from 7333509783866191872 to 7333509783721611264. Current Snapshot: MvccSnapshot[applied={T|T < 7333509783866191872}]
I20260926 07:27:42.107707 6749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:42.107961 6749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:42.111429 6749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcc7a38eeb694eb091b84d219b35ec86: Sending DeleteTablet for 3 replicas of tablet e04b90e259224eb58707f0ba494df26c
I20260926 07:27:42.112465 7138 tablet_service.cc:1514] Processing DeleteTablet for tablet e04b90e259224eb58707f0ba494df26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:42 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:45692
I20260926 07:27:42.112525 7006 tablet_service.cc:1514] Processing DeleteTablet for tablet e04b90e259224eb58707f0ba494df26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:42 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:59148
I20260926 07:27:42.112692 6873 tablet_service.cc:1514] Processing DeleteTablet for tablet e04b90e259224eb58707f0ba494df26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:42 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:38016
I20260926 07:27:42.114804 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6811
I20260926 07:27:42.124869 7244 tablet_replica.cc:330] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: stopping tablet replica
I20260926 07:27:42.125447 7244 raft_consensus.cc:2229] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:42.126058 7244 raft_consensus.cc:2258] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:42.128170 7244 ts_tablet_manager.cc:1905] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:42.135062 7244 ts_tablet_manager.cc:1918] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.135308 7244 log.cc:1196] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/e04b90e259224eb58707f0ba494df26c
I20260926 07:27:42.135919 7244 ts_tablet_manager.cc:1939] T e04b90e259224eb58707f0ba494df26c P 0db74dc71b3941b29f2fe99ecf5b3674: Deleting consensus metadata
I20260926 07:27:42.137742 6736 catalog_manager.cc:4841] TS 0db74dc71b3941b29f2fe99ecf5b3674 (127.2.147.195:40297): tablet e04b90e259224eb58707f0ba494df26c (table test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]) successfully deleted
I20260926 07:27:42.148471 7245 tablet_replica.cc:330] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: stopping tablet replica
I20260926 07:27:42.149051 7245 raft_consensus.cc:2229] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:42.149674 7245 raft_consensus.cc:2258] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:42.150615 7246 tablet_replica.cc:330] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: stopping tablet replica
I20260926 07:27:42.151115 7246 raft_consensus.cc:2229] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:42.151813 7246 raft_consensus.cc:2258] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:42.152740 7245 ts_tablet_manager.cc:1905] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:42.153887 7246 ts_tablet_manager.cc:1905] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:27:42 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:42.160682 7245 ts_tablet_manager.cc:1918] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.160928 7245 log.cc:1196] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/e04b90e259224eb58707f0ba494df26c
I20260926 07:27:42.161123 7246 ts_tablet_manager.cc:1918] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.161299 7246 log.cc:1196] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/e04b90e259224eb58707f0ba494df26c
I20260926 07:27:42.161609 7245 ts_tablet_manager.cc:1939] T e04b90e259224eb58707f0ba494df26c P 59404dce922d4260a5eea7022e6fcebc: Deleting consensus metadata
I20260926 07:27:42.161814 7246 ts_tablet_manager.cc:1939] T e04b90e259224eb58707f0ba494df26c P 3710657d71814e9a957d8991c04a0f53: Deleting consensus metadata
I20260926 07:27:42.163028 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.117682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.2.147.193:38325)
0926 07:27:42.117899 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.117908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.141810 (+ 23902us) server_negotiation.cc:407] Connection header received
0926 07:27:42.142395 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.142405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.142515 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.142608 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.144863 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.146522 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.149045 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.149843 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.149969 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.155285 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.155298 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.155314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.155361 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.159660 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.160470 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.160481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.160487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.160657 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.161122 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.161129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.161133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.161596 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.161882 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.162433 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.162668 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:27:42.163416 6734 catalog_manager.cc:4841] TS 3710657d71814e9a957d8991c04a0f53 (127.2.147.193:38325): tablet e04b90e259224eb58707f0ba494df26c (table test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]) successfully deleted
I20260926 07:27:42.163635 6736 catalog_manager.cc:4841] TS 59404dce922d4260a5eea7022e6fcebc (127.2.147.194:37643): tablet e04b90e259224eb58707f0ba494df26c (table test-table [id=d428c0f22ccd4baeb9c33dd3c740f2cd]) successfully deleted
I20260926 07:27:42.164660 6933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:42.265514 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6811
I20260926 07:27:42.292969 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6944
Sep 26 07:27:42 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:42.308398 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.293747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.2.147.194:37643)
0926 07:27:42.293924 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.293929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.293996 (+ 67us) server_negotiation.cc:407] Connection header received
0926 07:27:42.294164 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.294168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.294225 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.294310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.295838 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.296737 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.298469 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.298847 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.298932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.303198 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.303242 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.303264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.303310 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.306082 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.306604 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.306610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.306612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.306723 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.307044 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.307047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.307049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.307447 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.307642 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.307885 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.308025 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:27:42.308977 7066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:42.398384 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6944
I20260926 07:27:42.414404 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7076
Sep 26 07:27:42 dist-test-slave-1dwd krb5kdc[6706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:42.428843 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.415158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.2.147.195:40297)
0926 07:27:42.415361 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.415368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.415418 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:27:42.415630 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.415635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.415695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.415810 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.417149 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.417858 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.419038 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.419510 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.419645 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.423667 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.423679 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.423695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.423724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.426130 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.426723 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.426734 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.426742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.426858 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.427360 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.427368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.427374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.427751 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.428058 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.428341 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.428541 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:27:42.429474 7198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:42.516343 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7076
I20260926 07:27:42.532388 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 6719
I20260926 07:27:42.533495 6780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:42.639789 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 6719
2026-09-26T07:27:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4099 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:42Z Disabled control of system clock
I20260926 07:27:42.703994 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43297
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:43297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:27:43.013175 7255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:43.013441 7255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:43.013525 7255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:43.020439 7255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:43.020530 7255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:43.020566 7255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:43.020620 7255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:43.020654 7255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:43.029214 7255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:43297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:43.030332 7255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:43.031915 7255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:43.043176 7260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.043262 7263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.043282 7261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.044106 7255 server_base.cc:1034] running on GCE node
I20260926 07:27:43.044860 7255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:43.046207 7255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:43.047385 7255 hybrid_clock.cc:648] HybridClock initialized: now 1790407663047352 us; error 32 us; skew 500 ppm
I20260926 07:27:43.047720 7255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:43.049834 7255 webserver.cc:460] Webserver started at http://127.2.147.254:39633/ using document root <none> and password file <none>
I20260926 07:27:43.050513 7255 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:43.050623 7255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:43.050948 7255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:43.053741 7255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "3ebc6b7e24ed46c3be5456029bbd93be"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "055dd63abf798459aec74e7459fe6cee"
server_key_iv: "fef1d7b66ec2a11112d62b297e07cd54"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.054514 7255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "3ebc6b7e24ed46c3be5456029bbd93be"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "055dd63abf798459aec74e7459fe6cee"
server_key_iv: "fef1d7b66ec2a11112d62b297e07cd54"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.059439 7255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:27:43.062522 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.063897 7255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:43.064078 7255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "3ebc6b7e24ed46c3be5456029bbd93be"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "055dd63abf798459aec74e7459fe6cee"
server_key_iv: "fef1d7b66ec2a11112d62b297e07cd54"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.064268 7255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:43.069510 7255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:43.070241 7255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:43.070523 7255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:43.087757 7255 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:43297
I20260926 07:27:43.087800 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:43297 every 8 connection(s)
I20260926 07:27:43.088965 7255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:43.092442 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:43.099254 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7255
I20260926 07:27:43.099438 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:43.100041 2639 external_mini_cluster.cc:1428] Setting key 2f77fc109553ae7384ed645e73d446c4
I20260926 07:27:43.102530 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: Bootstrap starting.
I20260926 07:27:43.107296 7321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:43.108781 7321 log.cc:824] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:43.110409 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.101960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.2.147.254:43297)
0926 07:27:43.103118 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:43.103133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:43.103192 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:27:43.106150 (+ 2958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:43.106183 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:43.106771 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:43.107241 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:43.108356 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:43.108376 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:43.108391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:43.108410 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:43.108506 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:43.108552 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:43.108734 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:43.108965 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 07:27:43.109219 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":313,"threads_started":1}
I20260926 07:27:43.112969 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: No bootstrap required, opened a new log
I20260926 07:27:43.120487 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } }
I20260926 07:27:43.120885 7321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:43.121033 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebc6b7e24ed46c3be5456029bbd93be, State: Initialized, Role: FOLLOWER
I20260926 07:27:43.121618 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } }
I20260926 07:27:43.121862 7321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:43.122004 7321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:43.122172 7321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:43.124372 7321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } }
I20260926 07:27:43.124843 7321 leader_election.cc:302] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ebc6b7e24ed46c3be5456029bbd93be; no voters:
I20260926 07:27:43.125406 7321 leader_election.cc:288] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:43.125536 7326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:43.126297 7326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [term 1 LEADER]: Becoming Leader. State: Replica: 3ebc6b7e24ed46c3be5456029bbd93be, State: Running, Role: LEADER
I20260926 07:27:43.126839 7326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } }
I20260926 07:27:43.127451 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:43.129663 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ebc6b7e24ed46c3be5456029bbd93be. Latest consensus state: current_term: 1 leader_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } } }
I20260926 07:27:43.130115 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:43.129918 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebc6b7e24ed46c3be5456029bbd93be" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43297 } } }
I20260926 07:27:43.130247 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:43.131157 7334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:43.136621 7334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:43.144615 7334 catalog_manager.cc:1347] Generated new cluster ID: 76e6ada88f494d09acca1559aec98859
I20260926 07:27:43.144738 7334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:43.155161 7334 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:43.155633 7334 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:43.168646 7334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: Generated new TSK 0
I20260926 07:27:43.169726 7334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:43.181730 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--builtin_ntp_servers=127.2.147.212:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:27:43.496742 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:43.497001 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:43.497162 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:43.504237 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:43.504317 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:43.504520 7345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:43.513027 7345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:43.514236 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:43.515990 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:43.529654 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.530599 7353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.531675 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.533622 7345 server_base.cc:1034] running on GCE node
I20260926 07:27:43.534150 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:43.534992 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:43.536206 7345 hybrid_clock.cc:648] HybridClock initialized: now 1790407663536156 us; error 58 us; skew 500 ppm
I20260926 07:27:43.536515 7345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:43.538775 7345 webserver.cc:460] Webserver started at http://127.2.147.193:44229/ using document root <none> and password file <none>
I20260926 07:27:43.539489 7345 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:43.539639 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:43.539943 7345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:43.542692 7345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "afdda6317eea451ba86fd0a222f00cc0"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "1ad8b52973400b7cfd559143de435e9f"
server_key_iv: "d12c380f40246298896392acadfcc5e2"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.543511 7345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "afdda6317eea451ba86fd0a222f00cc0"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "1ad8b52973400b7cfd559143de435e9f"
server_key_iv: "d12c380f40246298896392acadfcc5e2"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.548254 7345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:27:43.551551 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.552758 7345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:27:43.552932 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "afdda6317eea451ba86fd0a222f00cc0"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "1ad8b52973400b7cfd559143de435e9f"
server_key_iv: "d12c380f40246298896392acadfcc5e2"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.553164 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:43.558701 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:43.559499 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:43.559764 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:43.560611 7345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:43.562510 7345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:43.562623 7345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.562754 7345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:43.562860 7345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 07:27:43.601366 7345 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38991
I20260926 07:27:43.601411 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38991 every 8 connection(s)
I20260926 07:27:43.603067 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:43.611779 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.605944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:36213 (local address 127.2.147.254:43297)
0926 07:27:43.606138 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:43.606143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:43.607734 (+ 1591us) server_negotiation.cc:407] Connection header received
0926 07:27:43.609262 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:43.609266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:43.609315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:43.609431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:43.611034 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:43.611039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:43.611042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:43.611056 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:43.611085 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:43.611097 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:43.611270 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:43.611457 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:27:43.611521 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:27:43.612293 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7345
I20260926 07:27:43.612494 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:43.612768 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.606592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43297 (local address 127.2.147.193:36213)
0926 07:27:43.607572 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.607614 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:43.609033 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:43.609656 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:43.609704 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:43.609828 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:43.610719 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:43.610722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:43.610759 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:43.610837 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:43.611230 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:43.611240 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:43.611249 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:27:43.611537 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 07:27:43.611799 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":281,"threads_started":1}
I20260926 07:27:43.613238 2639 external_mini_cluster.cc:1428] Setting key 30f29f03596a2156d77fbb69f46974b5
I20260926 07:27:43.614315 7472 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43297
I20260926 07:27:43.614631 7472 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:43.615373 7472 heartbeater.cc:511] Master 127.2.147.254:43297 requested a full tablet report, sending...
I20260926 07:27:43.615847 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--builtin_ntp_servers=127.2.147.212:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:27:43.617656 7286 ts_manager.cc:194] Registered new tserver with Master: afdda6317eea451ba86fd0a222f00cc0 (127.2.147.193:38991)
W20260926 07:27:43.929539 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:43.929845 7475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:43.930045 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:43.937525 7475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:43.937655 7475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:43.937839 7475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:43.946858 7475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:43.948189 7475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:43.949954 7475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:43.963347 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.963449 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.963428 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.964116 7475 server_base.cc:1034] running on GCE node
I20260926 07:27:43.964761 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:43.965682 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:43.966886 7475 hybrid_clock.cc:648] HybridClock initialized: now 1790407663966835 us; error 42 us; skew 500 ppm
I20260926 07:27:43.967177 7475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:43.969571 7475 webserver.cc:460] Webserver started at http://127.2.147.194:40815/ using document root <none> and password file <none>
I20260926 07:27:43.970230 7475 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:43.970317 7475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:43.970584 7475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:43.973438 7475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "c6bc0b799232434da1f9f87034b51f19"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "8d7f30235c61462274d18b00b8eb1b48"
server_key_iv: "518ca2038cb401af46237f59ff709cdc"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.974236 7475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "c6bc0b799232434da1f9f87034b51f19"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "8d7f30235c61462274d18b00b8eb1b48"
server_key_iv: "518ca2038cb401af46237f59ff709cdc"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.979344 7475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 07:27:43.982733 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.984228 7475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:43.984355 7475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "c6bc0b799232434da1f9f87034b51f19"
format_stamp: "Formatted at 2026-09-26 07:27:43 on dist-test-slave-1dwd"
server_key: "8d7f30235c61462274d18b00b8eb1b48"
server_key_iv: "518ca2038cb401af46237f59ff709cdc"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.984542 7475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:43.989957 7475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:43.990633 7475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:43.990881 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:43.991897 7475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:43.993162 7475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:43.993245 7475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.993316 7475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:43.993362 7475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:44.025228 7475 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:39803
I20260926 07:27:44.025281 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:39803 every 8 connection(s)
I20260926 07:27:44.026547 7475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:44.034585 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.029305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:52579 (local address 127.2.147.254:43297)
0926 07:27:44.029534 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.029541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.030999 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 07:27:44.032685 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.032689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.032742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.032831 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.034026 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.034030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.034033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.034047 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.034076 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.034087 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.034189 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.034334 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.034381 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:27:44.035140 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7475
I20260926 07:27:44.035326 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:44.035624 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.029901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43297 (local address 127.2.147.194:52579)
0926 07:27:44.030786 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.030879 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.032445 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.033037 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.033046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.033150 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.033761 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.033763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.033798 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.033877 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.034168 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.034175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.034180 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.034377 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.034558 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":280,"threads_started":1}
I20260926 07:27:44.035764 2639 external_mini_cluster.cc:1428] Setting key a7551a09764b6c085efba12a92c13162
I20260926 07:27:44.036983 7602 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43297
I20260926 07:27:44.037298 7602 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:44.037899 7602 heartbeater.cc:511] Master 127.2.147.254:43297 requested a full tablet report, sending...
I20260926 07:27:44.038331 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--builtin_ntp_servers=127.2.147.212:43691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:27:44.039309 7286 ts_manager.cc:194] Registered new tserver with Master: c6bc0b799232434da1f9f87034b51f19 (127.2.147.194:39803)
W20260926 07:27:44.353220 7605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:44.353473 7605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:44.353617 7605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:44.360823 7605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:44.360908 7605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:44.361136 7605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:44.369956 7605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:44.371162 7605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:44.372927 7605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:44.385315 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:44.385371 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:44.385401 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:44.386233 7605 server_base.cc:1034] running on GCE node
I20260926 07:27:44.386790 7605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:44.387738 7605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:44.388957 7605 hybrid_clock.cc:648] HybridClock initialized: now 1790407664388889 us; error 64 us; skew 500 ppm
I20260926 07:27:44.389319 7605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:44.391454 7605 webserver.cc:460] Webserver started at http://127.2.147.195:39349/ using document root <none> and password file <none>
I20260926 07:27:44.392127 7605 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:44.392262 7605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:44.392560 7605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:44.395326 7605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "0caae0438b284e5badee69b89d693b9e"
format_stamp: "Formatted at 2026-09-26 07:27:44 on dist-test-slave-1dwd"
server_key: "9e8e602910af09ba550881e14c925664"
server_key_iv: "3bacd02283fdd1b19612aa2e7510da90"
server_key_version: "encryptionkey@0"
I20260926 07:27:44.396116 7605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "0caae0438b284e5badee69b89d693b9e"
format_stamp: "Formatted at 2026-09-26 07:27:44 on dist-test-slave-1dwd"
server_key: "9e8e602910af09ba550881e14c925664"
server_key_iv: "3bacd02283fdd1b19612aa2e7510da90"
server_key_version: "encryptionkey@0"
I20260926 07:27:44.400807 7605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:27:44.403942 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:44.405128 7605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:27:44.405283 7605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "0caae0438b284e5badee69b89d693b9e"
format_stamp: "Formatted at 2026-09-26 07:27:44 on dist-test-slave-1dwd"
server_key: "9e8e602910af09ba550881e14c925664"
server_key_iv: "3bacd02283fdd1b19612aa2e7510da90"
server_key_version: "encryptionkey@0"
I20260926 07:27:44.405510 7605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:44.410682 7605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:44.411386 7605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:44.411587 7605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:44.412345 7605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:44.413601 7605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:44.413683 7605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:44.413757 7605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:44.413810 7605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:44.445132 7605 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:35165
I20260926 07:27:44.445175 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:35165 every 8 connection(s)
I20260926 07:27:44.446475 7605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:44.454432 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.448925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43725 (local address 127.2.147.254:43297)
0926 07:27:44.449104 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.449109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.450438 (+ 1329us) server_negotiation.cc:407] Connection header received
0926 07:27:44.452236 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.452241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.452309 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.452415 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.453695 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.453700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.453705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.453723 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.453761 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.453772 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.453891 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.454072 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.454136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:44.455440 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.449444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43297 (local address 127.2.147.195:43725)
0926 07:27:44.450297 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.450331 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.451966 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.452632 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.452639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.452731 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.453418 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.453420 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.453455 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.453533 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.453863 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.453873 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.453880 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.454151 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.454418 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":252,"threads_started":1}
I20260926 07:27:44.456661 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7605
I20260926 07:27:44.456804 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:44.456893 7732 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43297
I20260926 07:27:44.457186 7732 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:44.457362 2639 external_mini_cluster.cc:1428] Setting key b4a44a033a8523907f22abcb66b87c4e
I20260926 07:27:44.457916 7732 heartbeater.cc:511] Master 127.2.147.254:43297 requested a full tablet report, sending...
I20260926 07:27:44.459338 7286 ts_manager.cc:194] Registered new tserver with Master: 0caae0438b284e5badee69b89d693b9e (127.2.147.195:35165)
I20260926 07:27:44.460343 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:44.466573 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.464402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.2.147.254:43297)
0926 07:27:44.464548 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.464552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.465053 (+ 501us) server_negotiation.cc:407] Connection header received
0926 07:27:44.465173 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.465176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.465226 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.465306 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.466116 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.466120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.466123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.466136 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.466164 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.466175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.466275 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.466339 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.466381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:27:44.470410 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:44.477036 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:44.505770 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.488750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35165 (local address 127.0.0.1:33582)
0926 07:27:44.501368 (+ 12618us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.501388 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.501486 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.502924 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.502936 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.503086 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.504064 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.504070 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.504121 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.504233 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.504801 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.504814 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.504826 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.505061 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.505176 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12474,"thread_start_us":12297,"threads_started":1}
I20260926 07:27:44.507289 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.488787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.2.147.195:35165)
0926 07:27:44.502148 (+ 13361us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.502161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.502207 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:27:44.502293 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.502305 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.502651 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.502848 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.504483 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.504497 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.504519 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.504545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.504620 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.504646 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.506615 (+ 1969us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.506685 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.506812 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13140,"thread_start_us":12941,"threads_started":1}
I20260926 07:27:44.508251 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.490105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38206)
0926 07:27:44.505155 (+ 15050us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.505173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.505282 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.506136 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.506141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.506162 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.506884 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.506889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.506909 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.506966 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.507654 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.507657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.507661 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.507818 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.507868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14927,"thread_start_us":14799,"threads_started":1}
I20260926 07:27:44.508199 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.490369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.2.147.193:38991)
0926 07:27:44.504720 (+ 14351us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.504732 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.505374 (+ 642us) server_negotiation.cc:407] Connection header received
0926 07:27:44.505431 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.505436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.505661 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.505804 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.507343 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.507366 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.507380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.507398 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.507460 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.507482 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.507600 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.507795 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.507896 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14144,"thread_start_us":14682,"threads_started":1}
I20260926 07:27:44.511162 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 (DEFAULT_TABLE table=test-table [id=1f2be27bc7274641b561f72873f6547d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:44.511837 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 (DEFAULT_TABLE table=test-table [id=1f2be27bc7274641b561f72873f6547d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:44.512737 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fc8d8f83b4403d9428ba5bb8f77f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:44.513132 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.488707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:39803 (local address 127.0.0.1:44264)
0926 07:27:44.489356 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.489394 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.489515 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.506976 (+ 17461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.506981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.507009 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.507726 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.507729 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.507744 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.507782 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.512322 (+ 4540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.512327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.512332 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.512653 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.512696 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":637,"threads_started":1}
I20260926 07:27:44.513176 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fc8d8f83b4403d9428ba5bb8f77f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:44.513903 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.489330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.2.147.194:39803)
0926 07:27:44.506145 (+ 16815us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.506155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.506184 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:27:44.506265 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.506275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.506484 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.507932 (+ 1448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.512084 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.512092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.512103 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.512126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.512176 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.512194 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.513531 (+ 1337us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.513595 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.513668 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16648,"thread_start_us":17183,"threads_started":1}
I20260926 07:27:44.516682 7537 tablet_service.cc:1467] Processing CreateTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 (DEFAULT_TABLE table=test-table [id=1f2be27bc7274641b561f72873f6547d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:44.517961 7537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fc8d8f83b4403d9428ba5bb8f77f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:44.529482 7750 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Bootstrap starting.
I20260926 07:27:44.529945 7751 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Bootstrap starting.
I20260926 07:27:44.530172 7752 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Bootstrap starting.
I20260926 07:27:44.533232 7752 tablet_bootstrap.cc:654] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:44.533488 7751 tablet_bootstrap.cc:654] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:44.534264 7752 log.cc:824] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:44.534502 7750 tablet_bootstrap.cc:654] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:44.534653 7751 log.cc:824] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:44.535634 7750 log.cc:824] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:44.537423 7752 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: No bootstrap required, opened a new log
I20260926 07:27:44.537612 7751 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: No bootstrap required, opened a new log
I20260926 07:27:44.537699 7752 ts_tablet_manager.cc:1397] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 07:27:44.537889 7751 ts_tablet_manager.cc:1397] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260926 07:27:44.538666 7750 tablet_bootstrap.cc:492] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: No bootstrap required, opened a new log
I20260926 07:27:44.538904 7750 ts_tablet_manager.cc:1397] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 07:27:44.545077 7751 raft_consensus.cc:348] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.545507 7751 raft_consensus.cc:374] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:44.545600 7751 raft_consensus.cc:729] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afdda6317eea451ba86fd0a222f00cc0, State: Initialized, Role: FOLLOWER
I20260926 07:27:44.546257 7751 consensus_queue.cc:260] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.547353 7472 heartbeater.cc:503] Master 127.2.147.254:43297 was elected leader, sending a full tablet report...
I20260926 07:27:44.547926 7751 ts_tablet_manager.cc:1428] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Time spent starting tablet: real 0.010s user 0.007s sys 0.005s
I20260926 07:27:44.548112 7752 raft_consensus.cc:348] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.548524 7752 raft_consensus.cc:374] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:44.548636 7752 raft_consensus.cc:729] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bc0b799232434da1f9f87034b51f19, State: Initialized, Role: FOLLOWER
I20260926 07:27:44.549201 7752 consensus_queue.cc:260] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.550220 7602 heartbeater.cc:503] Master 127.2.147.254:43297 was elected leader, sending a full tablet report...
I20260926 07:27:44.550889 7752 ts_tablet_manager.cc:1428] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 07:27:44.551905 7750 raft_consensus.cc:348] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.552322 7750 raft_consensus.cc:374] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:44.552420 7750 raft_consensus.cc:729] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0caae0438b284e5badee69b89d693b9e, State: Initialized, Role: FOLLOWER
I20260926 07:27:44.553049 7750 consensus_queue.cc:260] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.554059 7732 heartbeater.cc:503] Master 127.2.147.254:43297 was elected leader, sending a full tablet report...
I20260926 07:27:44.554663 7750 ts_tablet_manager.cc:1428] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260926 07:27:44.605381 7473 tablet.cc:2369] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:44.614660 7756 raft_consensus.cc:482] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:44.614888 7756 raft_consensus.cc:504] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.616467 7756 leader_election.cc:288] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0caae0438b284e5badee69b89d693b9e (127.2.147.195:35165), c6bc0b799232434da1f9f87034b51f19 (127.2.147.194:39803)
I20260926 07:27:44.619642 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.616824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:48137 (local address 127.2.147.195:35165)
0926 07:27:44.617008 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.617016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.617730 (+ 714us) server_negotiation.cc:407] Connection header received
0926 07:27:44.617829 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.617834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.617895 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.617990 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.619018 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.619024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.619028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.619046 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.619079 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.619097 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.619252 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.619312 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.619373 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:27:44.619818 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.616905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:54765 (local address 127.2.147.194:39803)
0926 07:27:44.617079 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.617088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.617861 (+ 773us) server_negotiation.cc:407] Connection header received
0926 07:27:44.617982 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.617986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.618042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.618160 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.619211 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.619233 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.619238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.619258 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.619293 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.619309 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.619420 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.619529 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.619585 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:44.620082 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.616930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:39803 (local address 127.2.147.193:54765)
0926 07:27:44.617738 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.617756 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.617857 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.618202 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.618205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.618224 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.619000 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.619003 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.619017 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.619055 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.619406 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.619410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.619415 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.619562 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.619605 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":487,"threads_started":1}
I20260926 07:27:44.620114 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.616884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35165 (local address 127.2.147.193:48137)
0926 07:27:44.617628 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.617643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.617735 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.618029 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.618033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.618054 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.618804 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:27:44.618808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.618827 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:27:44.618874 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.619194 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.619219 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.619225 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.619347 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.619397 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":415,"threads_started":1}
I20260926 07:27:44.621021 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fc8d8f83b4403d9428ba5bb8f77f71" candidate_uuid: "afdda6317eea451ba86fd0a222f00cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bc0b799232434da1f9f87034b51f19" is_pre_election: true
I20260926 07:27:44.621021 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fc8d8f83b4403d9428ba5bb8f77f71" candidate_uuid: "afdda6317eea451ba86fd0a222f00cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0caae0438b284e5badee69b89d693b9e" is_pre_election: true
I20260926 07:27:44.621522 7557 raft_consensus.cc:2454] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afdda6317eea451ba86fd0a222f00cc0 in term 0.
I20260926 07:27:44.621521 7687 raft_consensus.cc:2454] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afdda6317eea451ba86fd0a222f00cc0 in term 0.
I20260926 07:27:44.622143 7363 leader_election.cc:302] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0caae0438b284e5badee69b89d693b9e, afdda6317eea451ba86fd0a222f00cc0, c6bc0b799232434da1f9f87034b51f19; no voters:
I20260926 07:27:44.622550 7756 raft_consensus.cc:2781] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:44.622668 7756 raft_consensus.cc:482] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:44.622802 7756 raft_consensus.cc:3037] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:44.624884 7756 raft_consensus.cc:504] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.625423 7756 leader_election.cc:288] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 0caae0438b284e5badee69b89d693b9e (127.2.147.195:35165), c6bc0b799232434da1f9f87034b51f19 (127.2.147.194:39803)
I20260926 07:27:44.625825 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fc8d8f83b4403d9428ba5bb8f77f71" candidate_uuid: "afdda6317eea451ba86fd0a222f00cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0caae0438b284e5badee69b89d693b9e"
I20260926 07:27:44.625841 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fc8d8f83b4403d9428ba5bb8f77f71" candidate_uuid: "afdda6317eea451ba86fd0a222f00cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bc0b799232434da1f9f87034b51f19"
I20260926 07:27:44.625990 7687 raft_consensus.cc:3037] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:44.626020 7557 raft_consensus.cc:3037] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:44.628346 7557 raft_consensus.cc:2454] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afdda6317eea451ba86fd0a222f00cc0 in term 1.
I20260926 07:27:44.628351 7687 raft_consensus.cc:2454] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afdda6317eea451ba86fd0a222f00cc0 in term 1.
I20260926 07:27:44.628722 7361 leader_election.cc:302] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afdda6317eea451ba86fd0a222f00cc0, c6bc0b799232434da1f9f87034b51f19; no voters:
I20260926 07:27:44.629015 7756 raft_consensus.cc:2781] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:44.629387 7756 raft_consensus.cc:686] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 1 LEADER]: Becoming Leader. State: Replica: afdda6317eea451ba86fd0a222f00cc0, State: Running, Role: LEADER
I20260926 07:27:44.629805 7756 consensus_queue.cc:237] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } }
I20260926 07:27:44.633862 7285 catalog_manager.cc:5495] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to afdda6317eea451ba86fd0a222f00cc0 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "afdda6317eea451ba86fd0a222f00cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afdda6317eea451ba86fd0a222f00cc0" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38991 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:44.648862 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.647111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.2.147.193:38991)
0926 07:27:44.647302 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.647307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.647323 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:27:44.647445 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.647449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.647518 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.647606 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.648399 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.648403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.648406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.648421 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.648451 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.648462 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.648559 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.648622 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.648663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:27:44.656816 7557 raft_consensus.cc:1261] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 1 FOLLOWER]: Refusing update from remote peer afdda6317eea451ba86fd0a222f00cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:44.656816 7687 raft_consensus.cc:1261] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 1 FOLLOWER]: Refusing update from remote peer afdda6317eea451ba86fd0a222f00cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:44.657778 7761 consensus_queue.cc:1035] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0caae0438b284e5badee69b89d693b9e" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:44.658126 7756 consensus_queue.cc:1035] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bc0b799232434da1f9f87034b51f19" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:44.684314 7286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:44.684594 7286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:44.687815 7286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ebc6b7e24ed46c3be5456029bbd93be: Sending DeleteTablet for 3 replicas of tablet 22fc8d8f83b4403d9428ba5bb8f77f71
I20260926 07:27:44.688840 7537 tablet_service.cc:1514] Processing DeleteTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:44 UTC) from {username='slave'} at 127.0.0.1:44264
I20260926 07:27:44.688856 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:44 UTC) from {username='slave'} at 127.0.0.1:33582
I20260926 07:27:44.689670 7774 tablet_replica.cc:330] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: stopping tablet replica
I20260926 07:27:44.689949 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 22fc8d8f83b4403d9428ba5bb8f77f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:44 UTC) from {username='slave'} at 127.0.0.1:38206
I20260926 07:27:44.690121 7774 raft_consensus.cc:2229] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:44.690687 7774 raft_consensus.cc:2258] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:44.692389 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7345
I20260926 07:27:44.692797 7774 ts_tablet_manager.cc:1905] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:44.699777 7774 ts_tablet_manager.cc:1918] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:44.699821 7776 tablet_replica.cc:330] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: stopping tablet replica
I20260926 07:27:44.700356 7775 tablet_replica.cc:330] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: stopping tablet replica
I20260926 07:27:44.700326 7776 raft_consensus.cc:2229] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:44.700805 7775 raft_consensus.cc:2229] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:44.701056 7776 raft_consensus.cc:2258] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:44.701349 7775 raft_consensus.cc:2258] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:44.701732 7774 log.cc:1196] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/22fc8d8f83b4403d9428ba5bb8f77f71
I20260926 07:27:44.702289 7774 ts_tablet_manager.cc:1939] T 22fc8d8f83b4403d9428ba5bb8f77f71 P c6bc0b799232434da1f9f87034b51f19: Deleting consensus metadata
I20260926 07:27:44.703707 7775 ts_tablet_manager.cc:1905] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:44.704068 7271 catalog_manager.cc:4841] TS c6bc0b799232434da1f9f87034b51f19 (127.2.147.194:39803): tablet 22fc8d8f83b4403d9428ba5bb8f77f71 (table test-table [id=1f2be27bc7274641b561f72873f6547d]) successfully deleted
I20260926 07:27:44.708648 7776 ts_tablet_manager.cc:1905] T 22fc8d8f83b4403d9428ba5bb8f77f71 P afdda6317eea451ba86fd0a222f00cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:44.709645 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.693947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.2.147.193:38991)
0926 07:27:44.694108 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.694115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.707071 (+ 12956us) server_negotiation.cc:407] Connection header received
0926 07:27:44.707274 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.707280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.707346 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.707464 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.708639 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.708647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.708655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.708677 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.708757 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.708773 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.708879 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.709243 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.709303 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:27:44.710847 7467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:44.712534 7775 ts_tablet_manager.cc:1918] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:44.712715 7775 log.cc:1196] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/22fc8d8f83b4403d9428ba5bb8f77f71
I20260926 07:27:44.713356 7775 ts_tablet_manager.cc:1939] T 22fc8d8f83b4403d9428ba5bb8f77f71 P 0caae0438b284e5badee69b89d693b9e: Deleting consensus metadata
I20260926 07:27:44.715332 7273 catalog_manager.cc:4841] TS 0caae0438b284e5badee69b89d693b9e (127.2.147.195:35165): tablet 22fc8d8f83b4403d9428ba5bb8f77f71 (table test-table [id=1f2be27bc7274641b561f72873f6547d]) successfully deleted
I20260926 07:27:44.803265 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7345
W20260926 07:27:44.820981 7271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.147.193:38991 (error 108)
I20260926 07:27:44.821663 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7475
I20260926 07:27:44.822781 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.822342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38230)
0926 07:27:44.822526 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.822648 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:27:44.822983 7271 catalog_manager.cc:4584] TS afdda6317eea451ba86fd0a222f00cc0 (127.2.147.193:38991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22fc8d8f83b4403d9428ba5bb8f77f71: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
I20260926 07:27:44.824826 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.822430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.2.147.194:39803)
0926 07:27:44.822612 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.822617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.823127 (+ 510us) server_negotiation.cc:407] Connection header received
0926 07:27:44.823373 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.823377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.823456 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.823529 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.824282 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.824286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.824289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.824302 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.824330 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.824340 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.824433 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.824464 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.824504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:27:44.825461 7597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:44.876785 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.876248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38232)
0926 07:27:44.876486 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.876594 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 07:27:44.877904 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.877499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38240)
0926 07:27:44.877681 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.877772 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:27:44.911181 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7475
I20260926 07:27:44.927659 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7605
I20260926 07:27:44.930316 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.928419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.2.147.195:35165)
0926 07:27:44.928566 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.928571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.928645 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:27:44.928812 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.928815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.928927 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.929019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.929764 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.929768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.929771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:44.929784 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.929813 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:44.929824 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.929917 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.929976 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.930017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:27:44.930958 7727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:44.938434 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.938032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38256)
0926 07:27:44.938212 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.938300 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:27:44.939453 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.939122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38258)
0926 07:27:44.939299 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.939358 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:27:45.017097 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7605
I20260926 07:27:45.020012 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.019606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38262)
0926 07:27:45.019798 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.019885 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:27:45.021013 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.020676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38991 (local address 127.0.0.1:38272)
0926 07:27:45.020851 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.020912 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:27:45.033646 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7255
I20260926 07:27:45.034691 7316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:45.144143 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7255
2026-09-26T07:27:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2502 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:27:45.183856 2639 mini_oidc.cc:152] Starting JWKS server
I20260926 07:27:45.186350 2639 webserver.cc:460] Webserver started at https://127.0.0.1:44945/ using document root <none> and password file <none>
I20260926 07:27:45.186650 2639 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:27:45.187312 2639 webserver.cc:460] Webserver started at http://127.0.0.1:45537/ using document root <none> and password file <none>
2026-09-26T07:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:45Z Disabled control of system clock
I20260926 07:27:45.212281 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:36229
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:36229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/testchainca.cert with env {}
W20260926 07:27:45.523445 7788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:45.523810 7788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:45.523957 7788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:45.530979 7788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:45.531122 7788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:45.531229 7788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:45.531338 7788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:45.531422 7788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:27:45.531522 7788 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:27:45.531595 7788 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44945/jwks.json
I20260926 07:27:45.540920 7788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:36229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:45.542114 7788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:45.543390 7788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:45.554780 7794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:45.554845 7796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:45.554844 7793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:45.555698 7788 server_base.cc:1034] running on GCE node
I20260926 07:27:45.556430 7788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:45.557714 7788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:45.558926 7788 hybrid_clock.cc:648] HybridClock initialized: now 1790407665558887 us; error 54 us; skew 500 ppm
I20260926 07:27:45.559319 7788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:45.561491 7788 webserver.cc:460] Webserver started at http://127.2.147.254:40189/ using document root <none> and password file <none>
I20260926 07:27:45.562144 7788 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:45.562276 7788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:45.562566 7788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:45.565340 7788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "c03d9266952a485f8c48bfa12467910d"
format_stamp: "Formatted at 2026-09-26 07:27:45 on dist-test-slave-1dwd"
server_key: "a1ecb7c3aa983a8a413efc6f9dadc909"
server_key_iv: "c2ec84db1bd4ba71104046e1999f5fdd"
server_key_version: "encryptionkey@0"
I20260926 07:27:45.566094 7788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "c03d9266952a485f8c48bfa12467910d"
format_stamp: "Formatted at 2026-09-26 07:27:45 on dist-test-slave-1dwd"
server_key: "a1ecb7c3aa983a8a413efc6f9dadc909"
server_key_iv: "c2ec84db1bd4ba71104046e1999f5fdd"
server_key_version: "encryptionkey@0"
I20260926 07:27:45.570698 7788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 07:27:45.573907 7802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:45.575284 7788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:45.575464 7788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "c03d9266952a485f8c48bfa12467910d"
format_stamp: "Formatted at 2026-09-26 07:27:45 on dist-test-slave-1dwd"
server_key: "a1ecb7c3aa983a8a413efc6f9dadc909"
server_key_iv: "c2ec84db1bd4ba71104046e1999f5fdd"
server_key_version: "encryptionkey@0"
I20260926 07:27:45.575685 7788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:45.588502 7788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:45.589214 7788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:45.589507 7788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:45.606984 7788 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:36229
I20260926 07:27:45.607005 7853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:36229 every 8 connection(s)
I20260926 07:27:45.608351 7788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:45.611999 7854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:45.617739 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7788
I20260926 07:27:45.617980 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:45.618592 2639 external_mini_cluster.cc:1428] Setting key 8bc69de980b210a06b14d645b787e323
I20260926 07:27:45.622207 7854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d: Bootstrap starting.
I20260926 07:27:45.625409 7854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:45.626626 7854 log.cc:824] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:45.629596 7854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d: No bootstrap required, opened a new log
I20260926 07:27:45.631907 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.620522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.2.147.254:36229)
0926 07:27:45.621486 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:45.621496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:45.621535 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:45.623163 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:45.623184 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:45.623700 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:45.624028 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:45.624641 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.625981 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.627788 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.628444 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.628574 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:45.629784 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:45.629790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:45.629803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:45.630295 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:45.630362 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:45.630380 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:45.630459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:45.630676 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 07:27:45.630972 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":356,"threads_started":1}
I20260926 07:27:45.635650 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:45.637969 7854 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } }
I20260926 07:27:45.638397 7854 raft_consensus.cc:374] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:45.638517 7854 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03d9266952a485f8c48bfa12467910d, State: Initialized, Role: FOLLOWER
I20260926 07:27:45.639071 7854 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } }
I20260926 07:27:45.639325 7854 raft_consensus.cc:388] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:45.639424 7854 raft_consensus.cc:482] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:45.639545 7854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:45.641556 7854 raft_consensus.cc:504] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } }
I20260926 07:27:45.641947 7854 leader_election.cc:302] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c03d9266952a485f8c48bfa12467910d; no voters:
I20260926 07:27:45.642602 7854 leader_election.cc:288] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:45.642720 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:45.643432 7859 raft_consensus.cc:686] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [term 1 LEADER]: Becoming Leader. State: Replica: c03d9266952a485f8c48bfa12467910d, State: Running, Role: LEADER
I20260926 07:27:45.643935 7859 consensus_queue.cc:237] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } }
I20260926 07:27:45.644584 7854 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:45.646488 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c03d9266952a485f8c48bfa12467910d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } } }
I20260926 07:27:45.646463 7861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c03d9266952a485f8c48bfa12467910d. Latest consensus state: current_term: 1 leader_uuid: "c03d9266952a485f8c48bfa12467910d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c03d9266952a485f8c48bfa12467910d" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36229 } } }
I20260926 07:27:45.646904 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:45.646904 7861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:45.649456 7864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:45.654282 7864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:45.663136 7864 catalog_manager.cc:1347] Generated new cluster ID: a1ff8f7e1d7b476885ad7da52bfce697
I20260926 07:27:45.663280 7864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:45.679642 7864 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:45.681461 7864 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:45.700259 7864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c03d9266952a485f8c48bfa12467910d: Generated new TSK 0
I20260926 07:27:45.701273 7864 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:27:45.737998 7884 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:27:45.738153 7884 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:27:45.739287 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.718111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.2.147.254:36229)
0926 07:27:45.718312 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:45.718319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:45.718767 (+ 448us) server_negotiation.cc:407] Connection header received
0926 07:27:45.719025 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:45.719035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:45.719132 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:45.719298 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:27:45.719779 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.720978 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.722372 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.723021 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.723128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:45.723643 (+ 515us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:27:45.738645 (+ 15002us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:27:45.738975 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 07:27:45.739409 7857 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:27:45.740624 7883 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:45.717982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:36229 (local address 127.0.0.1:41908)
0926 07:27:45.718674 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.718689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:45.718819 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:45.719353 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:45.719356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:45.719365 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:27:45.719646 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:45.719654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.721106 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.721109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:45.722232 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:45.722242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:45.723178 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:45.723182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:45.723342 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:45.723391 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:27:45.738889 (+ 15498us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:27:45.739134 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.147.254:36229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":382,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.147.254:36229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:27:45.742290 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7788
I20260926 07:27:45.743402 7849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:45.833412 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7788
2026-09-26T07:27:45Z chronyd exiting
I20260926 07:27:45.883107 2639 test_util.cc:175] -----------------------------------------------
I20260926 07:27:45.883318 2639 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407623164223-2639-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:27:45.886933 2639 mini_oidc.cc:152] Starting JWKS server
I20260926 07:27:45.888783 2639 webserver.cc:460] Webserver started at https://127.0.0.1:36653/ using document root <none> and password file <none>
I20260926 07:27:45.889011 2639 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:27:45.889652 2639 webserver.cc:460] Webserver started at http://127.0.0.1:39047/ using document root <none> and password file <none>
2026-09-26T07:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:45Z Disabled control of system clock
I20260926 07:27:45.914696 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43385
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:43385
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/testchainca.cert with env {}
W20260926 07:27:46.218185 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:46.218503 7891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:46.218667 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:46.226325 7891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:46.226456 7891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:46.226532 7891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:46.226643 7891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:46.226720 7891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:27:46.226809 7891 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:27:46.226863 7891 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36653/jwks.json
I20260926 07:27:46.236346 7891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:43385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:46.237455 7891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:46.238631 7891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:46.250075 7899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:46.250161 7896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:46.250123 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:46.250821 7891 server_base.cc:1034] running on GCE node
I20260926 07:27:46.251655 7891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:46.253006 7891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:46.254227 7891 hybrid_clock.cc:648] HybridClock initialized: now 1790407666254194 us; error 49 us; skew 500 ppm
I20260926 07:27:46.254561 7891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:46.256711 7891 webserver.cc:460] Webserver started at http://127.2.147.254:38999/ using document root <none> and password file <none>
I20260926 07:27:46.257377 7891 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:46.257511 7891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:46.257824 7891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:46.260625 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "4cd9d7d2818f42e8ae1810e10f313d7c"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "750897603bd6791b54b80021aa3bb4a9"
server_key_iv: "de71489a13d97b587aa84bc9f364e360"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.261394 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "4cd9d7d2818f42e8ae1810e10f313d7c"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "750897603bd6791b54b80021aa3bb4a9"
server_key_iv: "de71489a13d97b587aa84bc9f364e360"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.266088 7891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 07:27:46.269312 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:46.270538 7891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:27:46.270670 7891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "4cd9d7d2818f42e8ae1810e10f313d7c"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "750897603bd6791b54b80021aa3bb4a9"
server_key_iv: "de71489a13d97b587aa84bc9f364e360"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.270833 7891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:46.303714 7891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:46.304554 7891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:46.304828 7891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:46.322294 7891 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:43385
I20260926 07:27:46.322324 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:43385 every 8 connection(s)
I20260926 07:27:46.323570 7891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:46.327107 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:46.330628 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7891
I20260926 07:27:46.330821 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:46.331621 2639 external_mini_cluster.cc:1428] Setting key 5f22bd4a11fc53317e922a0b80119e83
I20260926 07:27:46.338966 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c: Bootstrap starting.
I20260926 07:27:46.341935 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:46.342980 7957 log.cc:824] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:46.344183 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:46.333392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.2.147.254:43385)
0926 07:27:46.334289 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:46.334299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:46.334336 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:27:46.335896 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:46.335920 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:46.336368 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:46.336731 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:46.337285 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.338477 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.339813 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.340485 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.340680 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:46.341803 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:46.341810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:46.341823 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:46.342397 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:46.342478 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:46.342497 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:46.342617 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:46.342835 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:27:46.343131 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":311,"threads_started":1}
I20260926 07:27:46.346516 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c: No bootstrap required, opened a new log
I20260926 07:27:46.348165 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:46.353870 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } }
I20260926 07:27:46.354176 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:46.354265 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd9d7d2818f42e8ae1810e10f313d7c, State: Initialized, Role: FOLLOWER
I20260926 07:27:46.354840 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } }
I20260926 07:27:46.355062 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:46.355207 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:46.355386 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:46.357641 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } }
I20260926 07:27:46.358161 7957 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd9d7d2818f42e8ae1810e10f313d7c; no voters:
I20260926 07:27:46.358688 7957 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:46.358837 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:46.359537 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [term 1 LEADER]: Becoming Leader. State: Replica: 4cd9d7d2818f42e8ae1810e10f313d7c, State: Running, Role: LEADER
I20260926 07:27:46.360030 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } }
I20260926 07:27:46.360634 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:46.362752 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd9d7d2818f42e8ae1810e10f313d7c. Latest consensus state: current_term: 1 leader_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } } }
I20260926 07:27:46.363584 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:46.363137 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd9d7d2818f42e8ae1810e10f313d7c" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43385 } } }
I20260926 07:27:46.364200 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:46.368080 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:46.372140 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:46.380167 7971 catalog_manager.cc:1347] Generated new cluster ID: 7f74ba0104d447fcbe74eb2410b762e4
I20260926 07:27:46.380281 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:46.417831 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:46.419639 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:46.426538 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd9d7d2818f42e8ae1810e10f313d7c: Generated new TSK 0
I20260926 07:27:46.427553 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:27:46.456915 7987 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:27:46.457029 7987 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:27:46.457942 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:46.438990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.2.147.254:43385)
0926 07:27:46.439239 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:46.439245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:46.439624 (+ 379us) server_negotiation.cc:407] Connection header received
0926 07:27:46.439870 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:46.439874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:46.439939 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:46.440056 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:27:46.440590 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.441257 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.443190 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.443761 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.443912 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:46.444404 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:27:46.457489 (+ 13085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:27:46.457751 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:27:46.458081 7960 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 07:27:46.458297 7986 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:46.438886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43385 (local address 127.0.0.1:56374)
0926 07:27:46.439519 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:46.439538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:46.439682 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:46.440108 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:46.440114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:46.440127 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:27:46.440427 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:46.440437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.441389 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.441394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:46.443050 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:46.443065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:46.443954 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:46.443961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:46.444132 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:46.444174 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:27:46.457681 (+ 13507us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:27:46.457916 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.147.254:43385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
I20260926 07:27:46.459898 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7891
I20260926 07:27:46.460923 7952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:46.550956 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7891
2026-09-26T07:27:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:27:46.605335 2639 mini_oidc.cc:152] Starting JWKS server
I20260926 07:27:46.607126 2639 webserver.cc:460] Webserver started at https://127.0.0.1:36565/ using document root <none> and password file <none>
I20260926 07:27:46.607399 2639 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:27:46.607980 2639 webserver.cc:460] Webserver started at http://127.0.0.1:41425/ using document root <none> and password file <none>
2026-09-26T07:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:46Z Disabled control of system clock
I20260926 07:27:46.632551 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41111
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36565/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:27:46.933429 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:46.933692 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:46.933771 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:46.940378 7994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:46.940466 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:46.940505 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:46.940550 7994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:46.940593 7994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:27:46.940649 7994 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:27:46.940686 7994 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36565/jwks.json
I20260926 07:27:46.948879 7994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36565/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:46.949961 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:46.951553 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:46.962808 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:46.962898 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:46.962867 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:46.964146 7994 server_base.cc:1034] running on GCE node
I20260926 07:27:46.964960 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:46.966328 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:46.967545 7994 hybrid_clock.cc:648] HybridClock initialized: now 1790407666967494 us; error 44 us; skew 500 ppm
I20260926 07:27:46.967877 7994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:46.975756 7994 webserver.cc:460] Webserver started at http://127.2.147.254:34493/ using document root <none> and password file <none>
I20260926 07:27:46.976429 7994 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:46.976558 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:46.976883 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:46.979719 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "79c666d295f544279a11fac254533ea5"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "0f9e146790cb47f47e8c46dfa30d7ffa"
server_key_iv: "2f9fb9007995f8b864b339be35e458fc"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.980481 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "79c666d295f544279a11fac254533ea5"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "0f9e146790cb47f47e8c46dfa30d7ffa"
server_key_iv: "2f9fb9007995f8b864b339be35e458fc"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.985385 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:46.988608 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:46.989884 7994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:27:46.990067 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "79c666d295f544279a11fac254533ea5"
format_stamp: "Formatted at 2026-09-26 07:27:46 on dist-test-slave-1dwd"
server_key: "0f9e146790cb47f47e8c46dfa30d7ffa"
server_key_iv: "2f9fb9007995f8b864b339be35e458fc"
server_key_version: "encryptionkey@0"
I20260926 07:27:46.990267 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:47.006270 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:47.007066 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:47.007426 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:47.024802 7994 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41111
I20260926 07:27:47.024809 8059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41111 every 8 connection(s)
I20260926 07:27:47.026081 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:47.027608 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 7994
I20260926 07:27:47.027812 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:47.028403 2639 external_mini_cluster.cc:1428] Setting key 25b43e4dbae16dde54a66cf5892755d0
I20260926 07:27:47.029982 8060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:47.042521 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.030081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.2.147.254:41111)
0926 07:27:47.031076 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:47.031092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:47.031143 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:27:47.032989 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:47.033025 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:47.033599 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:47.034013 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:47.034523 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.036543 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.037775 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.038727 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.038887 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.039936 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:47.039946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:47.039969 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:47.040679 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:47.040765 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:47.040794 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:47.040913 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:47.041082 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:27:47.041391 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":317,"threads_started":1}
I20260926 07:27:47.043836 8060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5: Bootstrap starting.
I20260926 07:27:47.046142 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:47.047031 8060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:47.048161 8060 log.cc:824] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:47.051061 8060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5: No bootstrap required, opened a new log
I20260926 07:27:47.058077 8060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } }
I20260926 07:27:47.058357 8060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:47.058414 8060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c666d295f544279a11fac254533ea5, State: Initialized, Role: FOLLOWER
I20260926 07:27:47.058892 8060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } }
I20260926 07:27:47.059075 8060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:47.059159 8060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:47.059346 8060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:47.061527 8060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } }
I20260926 07:27:47.061892 8060 leader_election.cc:302] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c666d295f544279a11fac254533ea5; no voters:
I20260926 07:27:47.062305 8060 leader_election.cc:288] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:47.062469 8065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:47.063086 8065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 79c666d295f544279a11fac254533ea5, State: Running, Role: LEADER
I20260926 07:27:47.063576 8065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } }
I20260926 07:27:47.064100 8060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:47.066112 8066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c666d295f544279a11fac254533ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } } }
I20260926 07:27:47.066541 8066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:47.066540 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c666d295f544279a11fac254533ea5. Latest consensus state: current_term: 1 leader_uuid: "79c666d295f544279a11fac254533ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c666d295f544279a11fac254533ea5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41111 } } }
I20260926 07:27:47.067538 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:47.068360 8074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:47.072535 8074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:47.085412 8074 catalog_manager.cc:1347] Generated new cluster ID: cba3a2befafe4ca0be72378d2a3efcfc
I20260926 07:27:47.085526 8074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:47.111106 8074 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:47.112790 8074 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:47.125710 8074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c666d295f544279a11fac254533ea5: Generated new TSK 0
I20260926 07:27:47.126735 8074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:47.148720 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.141189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.2.147.254:41111)
0926 07:27:47.141392 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:47.141399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:47.141728 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:27:47.141921 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:47.141924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:47.141981 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:47.142084 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:27:47.142572 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.143246 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.145156 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.145568 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.145667 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.146262 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:27:47.148066 (+ 1804us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:27:47.148421 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 07:27:47.148835 8063 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:27:47.148927 8089 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:47.141112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41111 (local address 127.0.0.1:39258)
0926 07:27:47.141627 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:47.141649 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:47.141781 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:47.142119 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:47.142124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:47.142137 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:27:47.142434 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:47.142445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.143402 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.143407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:47.145029 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:47.145043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.145759 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.145766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:47.145984 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.146029 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:27:47.148304 (+ 2275us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:27:47.148523 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.147.254:41111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260926 07:27:47.149693 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 7994
I20260926 07:27:47.150694 8055 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 07:27:47.160645 8090 webserver.cc:533] Webserver: sslize failed:
E20260926 07:27:47.160722 8090 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:27:47.239526 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 7994
2026-09-26T07:27:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:47Z Disabled control of system clock
I20260926 07:27:47.318578 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46729
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46729
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/testchainca.cert with env {}
W20260926 07:27:47.624181 8095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:47.624460 8095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:47.624552 8095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:47.631285 8095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:47.631372 8095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:47.631405 8095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:47.631455 8095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:47.631494 8095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:27:47.631548 8095 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:27:47.631583 8095 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:27:47.640820 8095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:47.641892 8095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:47.643121 8095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:47.654534 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:47.654589 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:47.654551 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:47.655525 8095 server_base.cc:1034] running on GCE node
I20260926 07:27:47.656315 8095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:47.657521 8095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:47.658712 8095 hybrid_clock.cc:648] HybridClock initialized: now 1790407667658680 us; error 24 us; skew 500 ppm
I20260926 07:27:47.659001 8095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:27:47.661072 8095 webserver.cc:460] Webserver started at http://127.2.147.254:33569/ using document root <none> and password file <none>
I20260926 07:27:47.661679 8095 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:47.661777 8095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:47.662011 8095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:47.664695 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "3a95c6cb68254a68868863983cd6e5a8"
format_stamp: "Formatted at 2026-09-26 07:27:47 on dist-test-slave-1dwd"
server_key: "0f4d6585c9f634421643ee22c5c07fe6"
server_key_iv: "792290341006bb530b0ae05bca365faf"
server_key_version: "encryptionkey@0"
I20260926 07:27:47.665408 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "3a95c6cb68254a68868863983cd6e5a8"
format_stamp: "Formatted at 2026-09-26 07:27:47 on dist-test-slave-1dwd"
server_key: "0f4d6585c9f634421643ee22c5c07fe6"
server_key_iv: "792290341006bb530b0ae05bca365faf"
server_key_version: "encryptionkey@0"
I20260926 07:27:47.669984 8095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:27:47.673108 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:47.674297 8095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:47.674425 8095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "3a95c6cb68254a68868863983cd6e5a8"
format_stamp: "Formatted at 2026-09-26 07:27:47 on dist-test-slave-1dwd"
server_key: "0f4d6585c9f634421643ee22c5c07fe6"
server_key_iv: "792290341006bb530b0ae05bca365faf"
server_key_version: "encryptionkey@0"
I20260926 07:27:47.674585 8095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:47.692480 8095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:47.693174 8095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:47.693397 8095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:47.710723 8095 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:46729
I20260926 07:27:47.710736 8160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:46729 every 8 connection(s)
I20260926 07:27:47.711963 8095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:47.714233 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8095
I20260926 07:27:47.714411 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:47.715190 2639 external_mini_cluster.cc:1428] Setting key 25674fafe3dc1e683c69c408efea55cc
I20260926 07:27:47.716382 8161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:47.727949 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.716999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.2.147.254:46729)
0926 07:27:47.717906 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:47.717916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:47.717955 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:27:47.719385 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:47.719404 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:47.719840 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:47.720165 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:47.720690 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.721933 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.723713 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.724347 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.724486 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.725756 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:47.725763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:47.725777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:27:47.726311 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:47.726378 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:27:47.726408 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:47.726497 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:47.726727 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 07:27:47.727037 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260926 07:27:47.729367 8161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8: Bootstrap starting.
I20260926 07:27:47.731570 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:47.732539 8161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:47.733453 8161 log.cc:824] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:47.736200 8161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8: No bootstrap required, opened a new log
I20260926 07:27:47.742792 8161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } }
I20260926 07:27:47.743100 8161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:47.743237 8161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a95c6cb68254a68868863983cd6e5a8, State: Initialized, Role: FOLLOWER
I20260926 07:27:47.743832 8161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } }
I20260926 07:27:47.744050 8161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:47.744194 8161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:47.744359 8161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:47.746290 8161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } }
I20260926 07:27:47.746745 8161 leader_election.cc:302] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a95c6cb68254a68868863983cd6e5a8; no voters:
I20260926 07:27:47.747337 8161 leader_election.cc:288] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:47.747462 8166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:47.748211 8166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [term 1 LEADER]: Becoming Leader. State: Replica: 3a95c6cb68254a68868863983cd6e5a8, State: Running, Role: LEADER
I20260926 07:27:47.748704 8166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } }
I20260926 07:27:47.749294 8161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:47.751953 8168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a95c6cb68254a68868863983cd6e5a8. Latest consensus state: current_term: 1 leader_uuid: "3a95c6cb68254a68868863983cd6e5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } } }
I20260926 07:27:47.751909 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a95c6cb68254a68868863983cd6e5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a95c6cb68254a68868863983cd6e5a8" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46729 } } }
I20260926 07:27:47.752305 8168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:47.752338 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:47.756119 8174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:47.760764 8174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:47.767691 8174 catalog_manager.cc:1347] Generated new cluster ID: 5cc30f9bc98843c4b7bd1da1b250ecc2
I20260926 07:27:47.767789 8174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:47.786636 8174 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:47.787887 8174 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:47.795245 8174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a95c6cb68254a68868863983cd6e5a8: Generated new TSK 0
I20260926 07:27:47.796088 8174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:48.250180 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.816722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.2.147.254:46729)
0926 07:27:47.816929 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:47.816939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:47.817425 (+ 486us) server_negotiation.cc:407] Connection header received
0926 07:27:47.817676 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:47.817680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:47.817749 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:47.817863 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:27:47.818403 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.819111 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.821027 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.821467 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.821563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.822110 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:27:48.249432 (+427322us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:27:48.249850 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54206: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:27:48.250313 8164 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54206: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:27:48.250355 8190 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:47.816724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46729 (local address 127.0.0.1:54206)
0926 07:27:47.817310 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:47.817329 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:47.817476 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:47.817910 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:47.817915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:47.817928 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:27:47.818255 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:47.818266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.819256 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.819261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:47.820897 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:47.820911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:47.821637 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:47.821644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:47.821828 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:47.821875 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:27:48.249701 (+427826us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:27:48.249982 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.147.254:46729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":295,"threads_started":1}
I20260926 07:27:48.251575 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8095
I20260926 07:27:48.252557 8156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:48.351240 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8095
2026-09-26T07:27:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1094 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (391 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (389 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:49 dist-test-slave-1dwd krb5kdc[8205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:49 dist-test-slave-1dwd krb5kdc[8205](info): set up 2 sockets
Sep 26 07:27:49 dist-test-slave-1dwd krb5kdc[8205](info): commencing operation
krb5kdc: starting...
W20260926 07:27:50.195899 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.007s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:50 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:27:50.388671 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38807
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:50.695462 8218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:50.695775 8218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:50.695924 8218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:50.702904 8218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:50.703045 8218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:50.703164 8218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:50.703279 8218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:50.703354 8218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:50.713328 8218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:50.714633 8218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:50.716028 8218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:50.727492 8226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:50.727487 8224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:50.727483 8223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:50.728384 8218 server_base.cc:1034] running on GCE node
I20260926 07:27:50.729161 8218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:50.730553 8218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:50.731755 8218 hybrid_clock.cc:648] HybridClock initialized: now 1790407670731713 us; error 32 us; skew 500 ppm
I20260926 07:27:50.732111 8218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:50 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407670, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:50.735096 8218 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:27:50.736644 8218 webserver.cc:460] Webserver started at http://127.2.147.254:37597/ using document root <none> and password file <none>
I20260926 07:27:50.737315 8218 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:50.737457 8218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:50.737833 8218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:50.740756 8218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "6391ebbc00bb4da5aa0d515dcaa7365f"
format_stamp: "Formatted at 2026-09-26 07:27:50 on dist-test-slave-1dwd"
server_key: "dd75a022f15feed3cd957743487fe5dc"
server_key_iv: "88f15bfdecb7d93b9dec7a7b312d8a26"
server_key_version: "encryptionkey@0"
I20260926 07:27:50.741492 8218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "6391ebbc00bb4da5aa0d515dcaa7365f"
format_stamp: "Formatted at 2026-09-26 07:27:50 on dist-test-slave-1dwd"
server_key: "dd75a022f15feed3cd957743487fe5dc"
server_key_iv: "88f15bfdecb7d93b9dec7a7b312d8a26"
server_key_version: "encryptionkey@0"
I20260926 07:27:50.746475 8218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 07:27:50.749730 8233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:50.750870 8218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:50.751039 8218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "6391ebbc00bb4da5aa0d515dcaa7365f"
format_stamp: "Formatted at 2026-09-26 07:27:50 on dist-test-slave-1dwd"
server_key: "dd75a022f15feed3cd957743487fe5dc"
server_key_iv: "88f15bfdecb7d93b9dec7a7b312d8a26"
server_key_version: "encryptionkey@0"
I20260926 07:27:50.751233 8218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:50.777029 8218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:50.780663 8218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:50.780932 8218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:50.797880 8218 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38807
I20260926 07:27:50.797912 8284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38807 every 8 connection(s)
I20260926 07:27:50.799116 8218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:50.802723 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:50.807400 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8218
I20260926 07:27:50.807595 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:50.808172 2639 external_mini_cluster.cc:1428] Setting key f75f8a08db75c4f9e7bf5d696255cff6
I20260926 07:27:50.813187 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f: Bootstrap starting.
I20260926 07:27:50.816648 8285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:50.817795 8285 log.cc:824] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:50.821017 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f: No bootstrap required, opened a new log
Sep 26 07:27:50 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:50.828421 8285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } }
I20260926 07:27:50.828747 8285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:50.828856 8285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6391ebbc00bb4da5aa0d515dcaa7365f, State: Initialized, Role: FOLLOWER
I20260926 07:27:50.829466 8285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } }
I20260926 07:27:50.829689 8285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:50.829846 8285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:50.830034 8285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:50.830538 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:50.809987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.2.147.254:38807)
0926 07:27:50.810758 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:50.810768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:50.810810 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:27:50.812480 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:50.812501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:50.812857 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:50.813170 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:50.814892 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:50.816176 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:50.817992 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:50.818983 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:50.819135 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:50.823500 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:50.823510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:50.823527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:50.823560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:50.826517 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:50.827244 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:50.827254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:50.827263 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:50.827431 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:50.827812 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:50.827817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:50.827821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:50.828272 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:50.828550 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:50.828976 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 07:27:50.829393 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":364,"threads_started":1}
I20260926 07:27:50.832767 8285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } }
I20260926 07:27:50.833248 8285 leader_election.cc:302] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6391ebbc00bb4da5aa0d515dcaa7365f; no voters:
I20260926 07:27:50.833745 8285 leader_election.cc:288] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:50.833835 8290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:50.834565 8290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [term 1 LEADER]: Becoming Leader. State: Replica: 6391ebbc00bb4da5aa0d515dcaa7365f, State: Running, Role: LEADER
I20260926 07:27:50.835052 8290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } }
I20260926 07:27:50.835602 8285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:50.837872 8291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } } }
I20260926 07:27:50.837800 8292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6391ebbc00bb4da5aa0d515dcaa7365f. Latest consensus state: current_term: 1 leader_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6391ebbc00bb4da5aa0d515dcaa7365f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38807 } } }
I20260926 07:27:50.838306 8291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:50.838306 8292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:50.840672 8296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:50.844522 8296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:50.851975 8296 catalog_manager.cc:1347] Generated new cluster ID: 51c60d5d25644f4095b21672139a7b03
I20260926 07:27:50.852077 8296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:50.873590 8296 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:50.875134 8296 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:50.880834 8296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6391ebbc00bb4da5aa0d515dcaa7365f: Generated new TSK 0
I20260926 07:27:50.881642 8296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:27:50.930740 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:51.243829 8311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:51.244088 8311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:51.244181 8311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:51.251124 8311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:51.251299 8311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:51.251428 8311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:51.261475 8311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:51.262708 8311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:51.263981 8311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:51.276752 8316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:51.276793 8319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:51.276830 8317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:51.277649 8311 server_base.cc:1034] running on GCE node
I20260926 07:27:51.278162 8311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:51.279003 8311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:51.280191 8311 hybrid_clock.cc:648] HybridClock initialized: now 1790407671280139 us; error 42 us; skew 500 ppm
I20260926 07:27:51.280462 8311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:51 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:51.283414 8311 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:27:51.284837 8311 webserver.cc:460] Webserver started at http://127.2.147.193:43119/ using document root <none> and password file <none>
I20260926 07:27:51.285449 8311 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:51.285553 8311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:51.285796 8311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:51.288609 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "b40e0b79defe4fa6a2a9c5360116f672"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "548a6496ccaf5087949d4aa73c1f9777"
server_key_iv: "b56a93dca625adf7801d84d12220400f"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.289434 8311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "b40e0b79defe4fa6a2a9c5360116f672"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "548a6496ccaf5087949d4aa73c1f9777"
server_key_iv: "b56a93dca625adf7801d84d12220400f"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.294307 8311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 07:27:51.297498 8326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.298758 8311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:51.298911 8311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b40e0b79defe4fa6a2a9c5360116f672"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "548a6496ccaf5087949d4aa73c1f9777"
server_key_iv: "b56a93dca625adf7801d84d12220400f"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.299105 8311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:51.324754 8311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:51.328728 8311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:51.329021 8311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:51.330013 8311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:51.331377 8311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:51.331480 8311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.331625 8311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:51.331696 8311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.365691 8311 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:41677
I20260926 07:27:51.365734 8438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:41677 every 8 connection(s)
I20260926 07:27:51.366935 8311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:51.369237 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8311
I20260926 07:27:51.369432 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:51.370064 2639 external_mini_cluster.cc:1428] Setting key 7ea04ebce6857aadbeb7608d1635bd5d
Sep 26 07:27:51 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:51.388178 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.370426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50317 (local address 127.2.147.254:38807)
0926 07:27:51.370828 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:51.370834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:51.370991 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:27:51.373339 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:51.373343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:51.373402 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:51.373498 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:51.375530 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.376251 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.378663 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.379051 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.379139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:51.382690 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:51.382696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:51.382701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:51.382738 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:51.385286 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:51.385958 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:51.385964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:51.385969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:51.386080 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:51.386503 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:51.386507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:51.386512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:51.386770 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:51.386921 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:51.387621 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 07:27:51.387825 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":916,"threads_started":1}
I20260926 07:27:51.389113 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.370004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38807 (local address 127.2.147.193:50317)
0926 07:27:51.370833 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:51.370884 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:51.373062 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:51.373756 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:51.373770 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:51.374551 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:51.375366 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:51.375393 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.376410 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.376418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:51.378527 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:51.378549 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.379284 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.379290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:51.379484 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:51.380219 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:51.380252 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:51.382471 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:51.385421 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:51.385429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:51.385445 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:51.385834 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:51.386185 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:51.386191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:51.386195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:51.386371 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:51.386900 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:51.386907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:51.387276 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 07:27:51.387535 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 07:27:51.388038 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":480,"threads_started":1}
I20260926 07:27:51.390661 8439 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38807
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:27:51.390954 8439 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:51.391678 8439 heartbeater.cc:511] Master 127.2.147.254:38807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:27:51.393743 8250 ts_manager.cc:194] Registered new tserver with Master: b40e0b79defe4fa6a2a9c5360116f672 (127.2.147.193:41677)
I20260926 07:27:51.395126 8250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:50317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:27:51.418001 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:51.731875 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:51.732136 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:51.732226 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:51.739542 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:51.739635 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:51.739748 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:51.748970 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:51.750144 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:51.751435 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:51.763979 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:51.764014 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:51.764382 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:51.765015 8445 server_base.cc:1034] running on GCE node
I20260926 07:27:51.765527 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:51.766327 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:51.767526 8445 hybrid_clock.cc:648] HybridClock initialized: now 1790407671767451 us; error 60 us; skew 500 ppm
I20260926 07:27:51.767822 8445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:51 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:51.770769 8445 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:27:51.772218 8445 webserver.cc:460] Webserver started at http://127.2.147.194:35227/ using document root <none> and password file <none>
I20260926 07:27:51.772881 8445 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:51.773008 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:51.773310 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:51.776142 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "10aef91d46fb4a27b51ff66b1bffac79"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "14d73122f69fc30b73a2f744d633c52d"
server_key_iv: "29c170cfac294465e7f36144285c6a10"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.776887 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "10aef91d46fb4a27b51ff66b1bffac79"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "14d73122f69fc30b73a2f744d633c52d"
server_key_iv: "29c170cfac294465e7f36144285c6a10"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.781822 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 07:27:51.785220 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.786435 8445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:51.786608 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "10aef91d46fb4a27b51ff66b1bffac79"
format_stamp: "Formatted at 2026-09-26 07:27:51 on dist-test-slave-1dwd"
server_key: "14d73122f69fc30b73a2f744d633c52d"
server_key_iv: "29c170cfac294465e7f36144285c6a10"
server_key_version: "encryptionkey@0"
I20260926 07:27:51.786819 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:51.810365 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:51.814563 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:51.814841 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:51.815819 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:51.817039 8445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:51.817109 8445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.817270 8445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:51.817319 8445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:51.849829 8445 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:43799
I20260926 07:27:51.849944 8572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:43799 every 8 connection(s)
I20260926 07:27:51.851130 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:51.856143 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8445
I20260926 07:27:51.856305 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:51.856742 2639 external_mini_cluster.cc:1428] Setting key 3efd1b08dcb5e9215988dd6efc19ef07
Sep 26 07:27:51 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407671, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:51.870857 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.853705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:42929 (local address 127.2.147.254:38807)
0926 07:27:51.853886 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:51.853890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:51.855144 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 07:27:51.857171 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:51.857176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:51.857262 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:51.857379 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:51.859070 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.859828 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.861323 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.861733 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.861864 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:51.865438 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:51.865444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:51.865449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:51.865494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:51.868039 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:51.868737 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:51.868743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:51.868746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:51.868878 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:51.869292 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:51.869298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:51.869302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:51.869544 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:51.869697 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:51.870321 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 07:27:51.870528 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:27:51.871939 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.854243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38807 (local address 127.2.147.194:42929)
0926 07:27:51.854970 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:51.855017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:51.856905 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:51.857643 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:51.857653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:51.858346 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:51.858905 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:51.858921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.859976 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.859980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:51.861185 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:51.861196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:51.861944 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:51.861952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:51.862181 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:51.862861 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:51.862883 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:51.865237 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:51.868182 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:51.868190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:51.868205 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:51.868614 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:51.868985 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:51.868990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:51.868994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:51.869185 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:51.869670 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:51.869678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:51.869988 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:27:51.870244 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 07:27:51.870749 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":264,"threads_started":1}
I20260926 07:27:51.873504 8573 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38807
I20260926 07:27:51.873793 8573 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:51.874478 8573 heartbeater.cc:511] Master 127.2.147.254:38807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:27:51.875970 8250 ts_manager.cc:194] Registered new tserver with Master: 10aef91d46fb4a27b51ff66b1bffac79 (127.2.147.194:43799)
I20260926 07:27:51.876817 8250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:42929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:27:51.900202 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:52.208568 8578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:52.208813 8578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:52.208997 8578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:52.216182 8578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:52.216300 8578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:52.216485 8578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:52.226262 8578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:52.227620 8578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:52.228999 8578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:52.241580 8583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:52.241559 8586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:52.241644 8584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:52.242616 8578 server_base.cc:1034] running on GCE node
I20260926 07:27:52.243176 8578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:52.244055 8578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:52.245267 8578 hybrid_clock.cc:648] HybridClock initialized: now 1790407672245231 us; error 48 us; skew 500 ppm
I20260926 07:27:52.245571 8578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.252082 8578 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:27:52.253508 8578 webserver.cc:460] Webserver started at http://127.2.147.195:37429/ using document root <none> and password file <none>
I20260926 07:27:52.254253 8578 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:52.254398 8578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:52.254698 8578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:52.257728 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "9aad45f0ae0c451eb63feba0287933d2"
format_stamp: "Formatted at 2026-09-26 07:27:52 on dist-test-slave-1dwd"
server_key: "1e4024d754f12d178b12ff7a6593b84c"
server_key_iv: "c5015cad3f8637b917b78b27cf8f9bde"
server_key_version: "encryptionkey@0"
I20260926 07:27:52.258522 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "9aad45f0ae0c451eb63feba0287933d2"
format_stamp: "Formatted at 2026-09-26 07:27:52 on dist-test-slave-1dwd"
server_key: "1e4024d754f12d178b12ff7a6593b84c"
server_key_iv: "c5015cad3f8637b917b78b27cf8f9bde"
server_key_version: "encryptionkey@0"
I20260926 07:27:52.263477 8578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 07:27:52.266731 8593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:52.268105 8578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:52.268266 8578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "9aad45f0ae0c451eb63feba0287933d2"
format_stamp: "Formatted at 2026-09-26 07:27:52 on dist-test-slave-1dwd"
server_key: "1e4024d754f12d178b12ff7a6593b84c"
server_key_iv: "c5015cad3f8637b917b78b27cf8f9bde"
server_key_version: "encryptionkey@0"
I20260926 07:27:52.268458 8578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:52.282145 8578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:52.286510 8578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:52.286798 8578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:52.287802 8578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:52.289072 8578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:52.289172 8578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:52.289315 8578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:52.289390 8578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:52.321184 8578 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:36495
I20260926 07:27:52.321230 8705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:36495 every 8 connection(s)
I20260926 07:27:52.322446 8578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:52.328612 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8578
I20260926 07:27:52.328774 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:52.329219 2639 external_mini_cluster.cc:1428] Setting key 346a0efd7edb073da138d5504fb99266
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:27:52.341801 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.324950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:33601 (local address 127.2.147.254:38807)
0926 07:27:52.325109 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.325113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.326241 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 07:27:52.328082 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.328088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.328173 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.328279 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.329885 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.330728 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.332521 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.332927 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.333057 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.336249 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.336254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.336257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.336286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.338960 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.339691 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.339700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.339703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.339807 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.340234 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.340238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.340242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.340490 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.340640 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.341264 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.341466 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:27:52.342669 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.325403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38807 (local address 127.2.147.195:33601)
0926 07:27:52.326096 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:52.326137 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:52.327800 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:52.328523 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:52.328532 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:52.329132 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:52.329724 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:52.329741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.330872 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.330877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:52.332373 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:52.332384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.333142 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.333150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:52.333376 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.334033 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:52.334054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:52.336044 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:52.339097 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:52.339105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:52.339119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:52.339569 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:52.339908 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:52.339914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:52.339917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:52.340118 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:52.340616 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:52.340623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:52.340939 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 07:27:52.341180 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 07:27:52.341626 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260926 07:27:52.344061 8706 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38807
I20260926 07:27:52.344341 8706 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:52.344996 8706 heartbeater.cc:511] Master 127.2.147.254:38807 requested a full tablet report, sending...
I20260926 07:27:52.346266 8249 ts_manager.cc:194] Registered new tserver with Master: 9aad45f0ae0c451eb63feba0287933d2 (127.2.147.195:36495)
I20260926 07:27:52.347078 8249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:33601
I20260926 07:27:52.355839 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/test-admin.keytab.
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.383992 2639 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:27:52.388708 8713 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:52.386420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.2.147.193:41677)
0926 07:27:52.386994 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.387002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.387027 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:27:52.387082 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.387088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.387349 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.387477 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.388249 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47514: BlockingRecv error: recv got EOF from 127.0.0.1:47514 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":306,"threads_started":1}
W20260926 07:27:52.388821 2639 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.390352 2639 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.391783 2639 init.cc:295] Successfully reacquired a new kerberos TGT
I20260926 07:27:52.398473 8439 heartbeater.cc:503] Master 127.2.147.254:38807 was elected leader, sending a full tablet report...
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:52.406502 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.393228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.2.147.193:41677)
0926 07:27:52.393408 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.393416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.393727 (+ 311us) server_negotiation.cc:407] Connection header received
0926 07:27:52.393919 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.393923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.393975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.394084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.395028 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.395993 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.397173 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.397713 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.397855 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.401719 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.401729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.401743 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.401780 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.404420 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.404941 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.404948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.404950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.405061 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.405329 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.405332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.405334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.405621 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.405808 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.406020 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.406249 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:27:52.411420 8713 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:52.410144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.2.147.193:41677)
0926 07:27:52.410310 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.410315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.410389 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:27:52.410492 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.410495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.410545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.410611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.411201 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47530: BlockingRecv error: recv got EOF from 127.0.0.1:47530 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:27:52.411587 2639 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.413009 2639 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.414237 2639 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:52.426745 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.415408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.2.147.193:41677)
0926 07:27:52.415585 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.415593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.415855 (+ 262us) server_negotiation.cc:407] Connection header received
0926 07:27:52.416009 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.416020 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.416067 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.416159 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.417058 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.417700 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.418799 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.419176 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.419355 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.422565 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.422569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.422572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.422607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.424815 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.425339 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.425343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.425345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.425448 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.425753 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.425756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.425758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.425998 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.426174 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.426388 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.426527 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:27:52.430650 8713 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:52.429158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.2.147.193:41677)
0926 07:27:52.429346 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.429352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.429519 (+ 167us) server_negotiation.cc:407] Connection header received
0926 07:27:52.429679 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.429684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.429754 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.429844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.430451 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47546: BlockingRecv error: recv got EOF from 127.0.0.1:47546 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 07:27:52.430807 2639 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407623164223-2639-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.432081 2639 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:52.433255 2639 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:27:52.445578 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.434323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.2.147.193:41677)
0926 07:27:52.434495 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.434500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.434729 (+ 229us) server_negotiation.cc:407] Connection header received
0926 07:27:52.434808 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.434811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.434852 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.434925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.435834 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.436610 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.437779 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.438200 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.438283 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.441295 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.441299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.441301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.441325 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.443668 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.444168 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.444172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.444174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.444292 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.444596 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.444599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.444600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.444812 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.444990 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.445159 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.445319 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:27:52.446864 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8311
I20260926 07:27:52.457197 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.447513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.2.147.193:41677)
0926 07:27:52.447669 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.447676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.447977 (+ 301us) server_negotiation.cc:407] Connection header received
0926 07:27:52.448134 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.448140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.448188 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.448273 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.449046 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.449616 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.450685 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.451048 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.451162 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.452894 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.452898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.452902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.452927 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.455126 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.455681 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.455684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.455686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.455785 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.456098 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.456114 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.456116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.456326 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.456493 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.456748 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.456906 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:27:52.457719 8434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:52.536888 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8311
I20260926 07:27:52.553138 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8445
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:27:52.567590 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.554012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.2.147.194:43799)
0926 07:27:52.554546 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.554554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.554578 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:27:52.554638 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.554645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.554857 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.555123 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.556064 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.557078 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.558234 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.558742 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.558854 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.562363 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.562373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.562387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.562420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.565170 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.565769 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.565776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.565778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.565890 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.566192 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.566195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.566197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.566480 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.566706 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.566949 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.567131 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":266,"threads_started":1}
I20260926 07:27:52.568746 8568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:52.648000 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8445
I20260926 07:27:52.663462 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8578
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:27:52.678907 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.664421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.2.147.195:36495)
0926 07:27:52.665068 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:52.665075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:52.665103 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:27:52.665161 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:52.665168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:52.665393 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:52.665582 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:52.666601 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.667761 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.669016 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:52.669571 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:52.669703 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:52.673669 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:52.673679 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:52.673694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:52.673724 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:52.676310 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.676968 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.676975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.676977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.677088 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:52.677470 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:52.677478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:52.677480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:52.677784 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:52.678099 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:52.678349 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:27:52.678593 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":355,"threads_started":1}
I20260926 07:27:52.680241 8701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:52.762934 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8578
I20260926 07:27:52.778364 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8218
I20260926 07:27:52.779512 8280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:52.878816 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8218
2026-09-26T07:27:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3749 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8733](info): set up 2 sockets
Sep 26 07:27:52 dist-test-slave-1dwd krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260926 07:27:53.960120 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:27:54 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab.
Entry for principal oryx/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab.
I20260926 07:27:54.156373 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46117
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab
--principal=oryx/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:54.464020 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:54.464265 8746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:54.464344 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:54.470806 8746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:54.470891 8746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:54.470933 8746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:54.470979 8746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:54.471014 8746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:54.479983 8746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab
--principal=oryx/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:54.481174 8746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:54.482393 8746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:54.493500 8752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:54.493515 8751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:54.493507 8754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:54.494122 8746 server_base.cc:1034] running on GCE node
I20260926 07:27:54.494807 8746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:54.496165 8746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:54.497380 8746 hybrid_clock.cc:648] HybridClock initialized: now 1790407674497339 us; error 47 us; skew 500 ppm
I20260926 07:27:54.497689 8746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:54 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:54.500699 8746 init.cc:377] Logged in from keytab as oryx/127.2.147.254@KRBTEST.COM (short username oryx)
I20260926 07:27:54.502067 8746 webserver.cc:460] Webserver started at http://127.2.147.254:41529/ using document root <none> and password file <none>
I20260926 07:27:54.502663 8746 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:54.502753 8746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:54.502979 8746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:54.505702 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "5beffe5f32114bd492e0e472af2042d5"
format_stamp: "Formatted at 2026-09-26 07:27:54 on dist-test-slave-1dwd"
server_key: "96c7277af5709f392d32acfecbb9a006"
server_key_iv: "783d14694ef14fcdaca334d69eb9d964"
server_key_version: "encryptionkey@0"
I20260926 07:27:54.506384 8746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "5beffe5f32114bd492e0e472af2042d5"
format_stamp: "Formatted at 2026-09-26 07:27:54 on dist-test-slave-1dwd"
server_key: "96c7277af5709f392d32acfecbb9a006"
server_key_iv: "783d14694ef14fcdaca334d69eb9d964"
server_key_version: "encryptionkey@0"
I20260926 07:27:54.511065 8746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 07:27:54.514362 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:54.515937 8746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:54.516093 8746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "5beffe5f32114bd492e0e472af2042d5"
format_stamp: "Formatted at 2026-09-26 07:27:54 on dist-test-slave-1dwd"
server_key: "96c7277af5709f392d32acfecbb9a006"
server_key_iv: "783d14694ef14fcdaca334d69eb9d964"
server_key_version: "encryptionkey@0"
I20260926 07:27:54.516247 8746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:54.537189 8746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:54.540879 8746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:54.541103 8746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:54.558539 8746 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:46117
I20260926 07:27:54.558575 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:46117 every 8 connection(s)
I20260926 07:27:54.559934 8746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:54.562844 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8746
I20260926 07:27:54.563042 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:27:54.563854 2639 external_mini_cluster.cc:1428] Setting key bced0d50df5ab513071886d4e1938a2c
I20260926 07:27:54.563891 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:54.578599 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Bootstrap starting.
Sep 26 07:27:54 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:54.581848 8813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:54.582885 8813 log.cc:824] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:54.585999 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: No bootstrap required, opened a new log
I20260926 07:27:54.586704 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:54.565846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.2.147.254:46117)
0926 07:27:54.566618 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:54.566629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:54.566671 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:27:54.569003 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:54.569028 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:54.569406 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:54.569725 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:54.571451 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:54.572695 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:54.574505 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:54.575170 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:54.575365 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:54.579837 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:54.579844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:54.579856 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:54.579888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:54.582766 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:54.583625 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:54.583634 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:54.583647 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:54.583845 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:54.584189 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:54.584195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:54.584199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:54.584678 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:54.584908 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:54.585331 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 07:27:54.585653 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260926 07:27:54.593209 8813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } }
I20260926 07:27:54.593541 8813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:54.593647 8813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beffe5f32114bd492e0e472af2042d5, State: Initialized, Role: FOLLOWER
I20260926 07:27:54.594158 8813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } }
I20260926 07:27:54.594381 8813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:54.594520 8813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:54.594671 8813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:54.596752 8813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } }
I20260926 07:27:54.597250 8813 leader_election.cc:302] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5beffe5f32114bd492e0e472af2042d5; no voters:
I20260926 07:27:54.597811 8813 leader_election.cc:288] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:54.597937 8818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:54.598593 8818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5beffe5f32114bd492e0e472af2042d5, State: Running, Role: LEADER
I20260926 07:27:54.599092 8818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } }
I20260926 07:27:54.599813 8813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:54.601753 8819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } }
I20260926 07:27:54.601778 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beffe5f32114bd492e0e472af2042d5. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } }
I20260926 07:27:54.602131 8819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:54.602221 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:54.607892 8823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:54.614014 8823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:54.622325 8823 catalog_manager.cc:1347] Generated new cluster ID: 52ba795d1f8b47639a872d597d6e00c3
I20260926 07:27:54.622434 8823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:54.643814 8823 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:54.645035 8823 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:54.659855 8823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Generated new TSK 0
I20260926 07:27:54.660669 8823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
I20260926 07:27:54.718446 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:55.032274 8839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:55.032532 8839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:55.032691 8839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:55.039644 8839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:55.039765 8839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:55.039937 8839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:55.049171 8839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:55.050350 8839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:55.051621 8839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:55.064220 8844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:55.064246 8847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:55.064301 8845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:55.065016 8839 server_base.cc:1034] running on GCE node
I20260926 07:27:55.065606 8839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:55.066478 8839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:55.067679 8839 hybrid_clock.cc:648] HybridClock initialized: now 1790407675067647 us; error 46 us; skew 500 ppm
I20260926 07:27:55.068007 8839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:55 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407675, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:55.070951 8839 init.cc:377] Logged in from keytab as oryx/127.2.147.193@KRBTEST.COM (short username oryx)
I20260926 07:27:55.072285 8839 webserver.cc:460] Webserver started at http://127.2.147.193:39461/ using document root <none> and password file <none>
I20260926 07:27:55.072888 8839 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:55.072978 8839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:55.073197 8839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:55.075963 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "b98a232158b547f38305ad1fb7546748"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "ddb7c2a620bbada49e1fe69e4d8898cb"
server_key_iv: "44f0ebd4ede6150a9082fef879bcfb41"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.076656 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "b98a232158b547f38305ad1fb7546748"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "ddb7c2a620bbada49e1fe69e4d8898cb"
server_key_iv: "44f0ebd4ede6150a9082fef879bcfb41"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.081324 8839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:55.084347 8854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.085527 8839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:27:55.085649 8839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b98a232158b547f38305ad1fb7546748"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "ddb7c2a620bbada49e1fe69e4d8898cb"
server_key_iv: "44f0ebd4ede6150a9082fef879bcfb41"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.085811 8839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:55.099300 8839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:55.104038 8839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:55.104372 8839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:55.105574 8839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:55.107584 8839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:55.107679 8839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.107805 8839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:55.107865 8839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.139392 8839 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:46847
I20260926 07:27:55.139444 8966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:46847 every 8 connection(s)
I20260926 07:27:55.140704 8839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:55.147614 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8839
I20260926 07:27:55.147833 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:55.148592 2639 external_mini_cluster.cc:1428] Setting key f79de88c0a91878eb435ccb467a2b2e1
Sep 26 07:27:55 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407675, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:55.161031 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.143063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:55363 (local address 127.2.147.254:46117)
0926 07:27:55.143543 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:55.143550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:55.144448 (+ 898us) server_negotiation.cc:407] Connection header received
0926 07:27:55.146073 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:55.146077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:55.146139 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:55.146233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:55.147798 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.148911 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.150812 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.151467 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.151601 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:55.154767 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:55.154776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:55.154783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:55.154841 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:55.157856 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:55.158766 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:55.158774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:55.158779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:55.158891 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:55.159352 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:55.159358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:55.159363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:55.159666 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:55.159768 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:55.160613 (+ 845us) server_negotiation.cc:299] Negotiation successful
0926 07:27:55.160763 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":252,"threads_started":1}
I20260926 07:27:55.162510 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.143627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:55363)
0926 07:27:55.144286 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:55.144330 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:55.145849 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:55.146429 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:55.146436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:55.147009 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:55.147610 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:55.147626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.149029 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.149033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:55.150688 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:55.150698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.151652 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.151656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:55.151827 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:55.152495 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:55.152515 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:55.154549 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:55.157989 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:55.158001 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:55.158019 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:55.158654 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:55.158984 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:55.158991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:55.158996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:55.159227 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:55.159789 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:55.159799 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:55.160217 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 07:27:55.160498 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 07:27:55.161020 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":256,"threads_started":1}
I20260926 07:27:55.164243 8967 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:55.164549 8967 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:55.165385 8967 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:55.167848 8778 ts_manager.cc:194] Registered new tserver with Master: b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:55.169943 8778 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:55363
WARNING: no policy specified for oryx/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
I20260926 07:27:55.194214 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:55.503021 8973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:55.503328 8973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:55.503494 8973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:55.510566 8973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:55.510672 8973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:55.510850 8973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:55.520010 8973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:55.521199 8973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:55.522405 8973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:55.534360 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:55.534392 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:55.535145 8973 server_base.cc:1034] running on GCE node
W20260926 07:27:55.535415 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:55.536003 8973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:55.536870 8973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:55.538097 8973 hybrid_clock.cc:648] HybridClock initialized: now 1790407675538056 us; error 48 us; skew 500 ppm
I20260926 07:27:55.538388 8973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:55 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407675, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:55.541406 8973 init.cc:377] Logged in from keytab as oryx/127.2.147.194@KRBTEST.COM (short username oryx)
I20260926 07:27:55.542639 8973 webserver.cc:460] Webserver started at http://127.2.147.194:38321/ using document root <none> and password file <none>
I20260926 07:27:55.543310 8973 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:55.543428 8973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:55.543704 8973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:55.546309 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "425211964e7a4614a119f70d6f77fbc3"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "aba93cbfb132e905f4282fa3c3e962b2"
server_key_iv: "a6b4aa930569a6ae884c8a9ec63e1c6c"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.547024 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "425211964e7a4614a119f70d6f77fbc3"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "aba93cbfb132e905f4282fa3c3e962b2"
server_key_iv: "a6b4aa930569a6ae884c8a9ec63e1c6c"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.551913 8973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:27:55.555073 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.556330 8973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:27:55.556494 8973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "425211964e7a4614a119f70d6f77fbc3"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "aba93cbfb132e905f4282fa3c3e962b2"
server_key_iv: "a6b4aa930569a6ae884c8a9ec63e1c6c"
server_key_version: "encryptionkey@0"
I20260926 07:27:55.556684 8973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:55.584918 8973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:55.588682 8973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:55.588973 8973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:55.589906 8973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:55.591159 8973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:55.591279 8973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.591420 8973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:55.591483 8973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:55.622313 8973 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:33103
I20260926 07:27:55.622344 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:33103 every 8 connection(s)
I20260926 07:27:55.623581 8973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:55.630849 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 8973
I20260926 07:27:55.631019 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:55.631582 2639 external_mini_cluster.cc:1428] Setting key 818316959b18c32fde020589e9c34898
Sep 26 07:27:55 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407675, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:55.643466 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.626028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58755 (local address 127.2.147.254:46117)
0926 07:27:55.626205 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:55.626210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:55.627345 (+ 1135us) server_negotiation.cc:407] Connection header received
0926 07:27:55.629131 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:55.629136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:55.629224 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:55.629326 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:55.630719 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.631813 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.633690 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.634096 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.634182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:55.638104 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:55.638109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:55.638112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:55.638142 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:55.640673 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:55.641364 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:55.641370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:55.641378 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:55.641497 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:55.641881 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:55.641888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:55.641891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:55.642123 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:55.642276 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:55.642879 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 07:27:55.643103 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:55.644325 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.626472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:58755)
0926 07:27:55.627144 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:55.627185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:55.628873 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:55.629549 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:55.629557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:55.630028 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:55.630561 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:55.630577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.631981 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.631986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:55.633559 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:55.633574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:55.634304 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:55.634313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:55.634649 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:55.635417 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:55.635459 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:55.637913 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:55.640794 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:55.640802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:55.640815 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:55.641225 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:55.641599 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:55.641605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:55.641609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:55.641778 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:55.642241 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:55.642248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:55.642548 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 07:27:55.642792 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 07:27:55.643290 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260926 07:27:55.645637 9101 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:55.645931 9101 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:55.646720 9101 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:55.648375 8778 ts_manager.cc:194] Registered new tserver with Master: 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103)
I20260926 07:27:55.649627 8778 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:58755
WARNING: no policy specified for oryx/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
I20260926 07:27:55.677886 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:55.985183 9106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:55.985462 9106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:55.985639 9106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:55.993353 9106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:55.993487 9106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:55.993649 9106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:56.002976 9106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:56.004294 9106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:56.005501 9106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:56.018352 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.018386 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.018352 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:56.019178 9106 server_base.cc:1034] running on GCE node
I20260926 07:27:56.019804 9106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:56.020735 9106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:56.021940 9106 hybrid_clock.cc:648] HybridClock initialized: now 1790407676021890 us; error 50 us; skew 500 ppm
I20260926 07:27:56.022235 9106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:56.025349 9106 init.cc:377] Logged in from keytab as oryx/127.2.147.195@KRBTEST.COM (short username oryx)
I20260926 07:27:56.026743 9106 webserver.cc:460] Webserver started at http://127.2.147.195:46883/ using document root <none> and password file <none>
I20260926 07:27:56.027566 9106 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:56.027701 9106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:56.027989 9106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:56.030709 9106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "6576e30eaac644bfbc4a2a612b70b887"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "81489bac324c620f49496c2578a92cb8"
server_key_iv: "ee0aa6456709157e1a365ee2fc89a029"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.031524 9106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "6576e30eaac644bfbc4a2a612b70b887"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "81489bac324c620f49496c2578a92cb8"
server_key_iv: "ee0aa6456709157e1a365ee2fc89a029"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.036085 9106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:27:56.039115 9121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.040266 9106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:27:56.040416 9106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "6576e30eaac644bfbc4a2a612b70b887"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "81489bac324c620f49496c2578a92cb8"
server_key_iv: "ee0aa6456709157e1a365ee2fc89a029"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.040604 9106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:56.058320 9106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:56.062032 9106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:56.062325 9106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:56.063164 9106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:56.064392 9106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:56.064495 9106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.064638 9106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:56.064705 9106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.095484 9106 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:36619
I20260926 07:27:56.095521 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:36619 every 8 connection(s)
I20260926 07:27:56.096693 9106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:56.104862 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9106
I20260926 07:27:56.105044 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:56.105535 2639 external_mini_cluster.cc:1428] Setting key ab62b186186648256363460f52830692
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:56.115454 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.098972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:56757 (local address 127.2.147.254:46117)
0926 07:27:56.099126 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.099130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.100246 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 07:27:56.102008 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.102012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.102078 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.102164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.103678 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.104458 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.106563 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.107096 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.107261 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.110551 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.110555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.110558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.110587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.112978 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.113596 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.113600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.113603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.113708 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.114054 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.114056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.114058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.114312 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.114414 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.114992 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.115149 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:27:56.116367 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.099412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:56757)
0926 07:27:56.100087 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.100123 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.101771 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.102376 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.102383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.102904 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.103491 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.103540 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.104598 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.104603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.106426 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.106441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.107355 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.107362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.107610 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.108298 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.108325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.110397 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.113092 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.113097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.113107 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.113498 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.113791 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.113794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.113796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.113951 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.114404 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.114409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.114686 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.114913 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.115311 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":241,"threads_started":1}
I20260926 07:27:56.117681 9234 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:56.117970 9234 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:56.118698 9234 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:56.120313 8777 ts_manager.cc:194] Registered new tserver with Master: 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:27:56.121155 8777 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:56757
I20260926 07:27:56.132347 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.253@KRBTEST.COM" created.
I20260926 07:27:56.172837 8967 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.253.keytab.
Entry for principal oryx/127.2.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.253.keytab.
I20260926 07:27:56.178515 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:45863
--webserver_interface=127.2.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.253.keytab
--principal=oryx/127.2.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46117,127.2.147.253:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:56.489938 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:56.490250 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:56.490406 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:56.497476 9239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:56.497562 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:56.497606 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:56.497660 9239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:56.497696 9239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:56.506589 9239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46117,127.2.147.253:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.253.keytab
--principal=oryx/127.2.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
W20260926 07:27:56.506901 9239 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.147.254:46117,127.2.147.253:45863'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:27:56.507872 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:56.509122 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:56.519794 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.519896 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.519888 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:56.520601 9239 server_base.cc:1034] running on GCE node
I20260926 07:27:56.521240 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:56.522428 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:56.523624 9239 hybrid_clock.cc:648] HybridClock initialized: now 1790407676523575 us; error 29 us; skew 500 ppm
I20260926 07:27:56.523886 9239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:56.526728 9239 init.cc:377] Logged in from keytab as oryx/127.2.147.253@KRBTEST.COM (short username oryx)
I20260926 07:27:56.528069 9239 webserver.cc:460] Webserver started at http://127.2.147.253:43153/ using document root <none> and password file <none>
I20260926 07:27:56.528699 9239 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:56.528807 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:56.529093 9239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:56.531827 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/instance:
uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "87165de6f8fb4b9ae542a399a516719e"
server_key_iv: "a0194a1e11819c8a6c9a4c61bc715e76"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.532529 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal/instance:
uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "87165de6f8fb4b9ae542a399a516719e"
server_key_iv: "a0194a1e11819c8a6c9a4c61bc715e76"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.537328 9239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:27:56.540593 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.541844 9239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:27:56.542008 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "87165de6f8fb4b9ae542a399a516719e"
server_key_iv: "a0194a1e11819c8a6c9a4c61bc715e76"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.542208 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:56.565007 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:56.569916 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:56.570165 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:56.587311 9239 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.253:45863
I20260926 07:27:56.587345 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.253:45863 every 8 connection(s)
I20260926 07:27:56.588554 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
I20260926 07:27:56.592242 9306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:56.595801 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9239
I20260926 07:27:56.598868 9306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.253@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:56.618391 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.602701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.2.147.254:46117)
0926 07:27:56.602843 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.602847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.603575 (+ 728us) server_negotiation.cc:407] Connection header received
0926 07:27:56.603822 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.603828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.603896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.603973 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.605380 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.606514 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.608341 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.608984 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.609103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.612869 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.612874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.612877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.612948 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.615476 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.616145 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.616148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.616150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.616314 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.616702 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.616705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.616707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.617133 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.617270 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.617951 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.618168 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:27:56.619526 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.602735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41010)
0926 07:27:56.603422 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.603463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.603647 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.604055 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.604067 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.604894 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.605242 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.605257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.606627 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.606631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.608186 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.608198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.609168 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.609173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.609285 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.609896 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.609933 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.612682 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.615601 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.615608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.615615 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.616027 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.616418 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.616421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.616424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.616597 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.617274 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.617281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.617483 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.617821 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.618287 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":135,"spinlock_wait_cycles":5248,"thread_start_us":289,"threads_started":1}
I20260926 07:27:56.619547 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.597827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.2.147.253:45863)
0926 07:27:56.598595 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.598608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.598655 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:27:56.600908 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.600943 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.601399 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.601763 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.603323 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.605404 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.606918 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.607869 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.608053 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.612458 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.612471 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.612490 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.612536 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.615404 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.616009 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.616017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.616021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.616186 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.616560 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.616565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.616568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.617066 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.617338 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.617734 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.618192 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":557,"spinlock_wait_cycles":90624,"thread_start_us":336,"threads_started":1}
I20260926 07:27:56.621291 9306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.253@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:56.633767 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.622163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41936)
0926 07:27:56.622385 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.622401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.622489 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.622789 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.622792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.623065 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.623177 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.623184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.624614 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.624618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.625482 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.625500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.626396 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.626399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.626555 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.627119 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.627134 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.629148 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.631895 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.631898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.631900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.632216 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.632510 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.632513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.632515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.632629 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.633109 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.633112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.633212 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.633312 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.633506 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":18}
I20260926 07:27:56.634004 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.622392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.2.147.253:45863)
0926 07:27:56.622531 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.622536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.622554 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:56.622600 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.622604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.622674 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.622780 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.623330 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.624472 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.625622 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.626274 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.626404 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.629308 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.629313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.629316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.629344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.631794 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.632308 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.632312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.632314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.632425 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.632726 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.632747 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.632749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.633015 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.633127 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.633394 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.633612 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":48}
I20260926 07:27:56.638236 9306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Bootstrap starting.
I20260926 07:27:56.641098 9306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:56.642024 9306 log.cc:824] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:56.644549 9306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: No bootstrap required, opened a new log
I20260926 07:27:56.651564 9306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } }
I20260926 07:27:56.651865 9306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:56.651963 9306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c78ec5d8ab42c4b782a03ef8f3a497, State: Initialized, Role: FOLLOWER
I20260926 07:27:56.652493 9306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } }
I20260926 07:27:56.652799 9101 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
I20260926 07:27:56.653486 9312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } } }
I20260926 07:27:56.653895 9312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:56.654765 9306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:27:56.663539 9323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:56.663699 9323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:56.665429 9239 master_runner.cc:428] Detected that this master 38c78ec5d8ab42c4b782a03ef8f3a497 is joining an existing cluster
I20260926 07:27:56.665525 9239 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.147.253:45863
I20260926 07:27:56.686113 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.669930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41952)
0926 07:27:56.670690 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.670702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.670851 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.671152 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.671156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.671712 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.672066 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.672074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.674515 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.674525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.676040 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.676053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.677193 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.677200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.677384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.678059 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.678081 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.679140 (+ 1059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.683224 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.683230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.683233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.683851 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.684239 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.684244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.684248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.684451 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.685099 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.685104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.685237 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.685373 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.685630 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":62,"spinlock_wait_cycles":30464,"thread_start_us":377,"threads_started":1}
I20260926 07:27:56.686942 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.669421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.2.147.253:45863)
0926 07:27:56.669590 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.669601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.670772 (+ 1171us) server_negotiation.cc:407] Connection header received
0926 07:27:56.670971 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.670976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.671031 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.671143 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.673191 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.674320 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.676194 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.677007 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.677144 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.679389 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.679397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.679404 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.679441 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.683072 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.683989 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.683995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.684000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.684127 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.684583 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.684589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.684591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.684980 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.685114 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.686402 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.686610 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260926 07:27:56.690111 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.669461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41012)
0926 07:27:56.670112 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.670131 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.670272 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.670811 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.670833 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.671602 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.672049 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.672061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.673147 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.673155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.681614 (+ 8459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.681626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.682838 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.682845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.683027 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.683587 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.683606 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.684583 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.687861 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.687866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.687870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.688263 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.688662 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.688665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.688667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.688826 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.689322 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.689325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.689508 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.689616 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.689794 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":348,"threads_started":1}
I20260926 07:27:56.690307 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.669392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.2.147.254:46117)
0926 07:27:56.669581 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.669587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.670220 (+ 633us) server_negotiation.cc:407] Connection header received
0926 07:27:56.670436 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.670443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.670531 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.670617 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.672198 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.673007 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.681787 (+ 8780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.682455 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.682553 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.684785 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.684792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.684796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.684830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.687732 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.688402 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.688414 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.688417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.688560 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.688936 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.688941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.688943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.689226 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.689352 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.689771 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.689993 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.2.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.252@KRBTEST.COM" created.
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.254@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:56.714485 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.696185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41958)
0926 07:27:56.699807 (+ 3622us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.699870 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.700045 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.700631 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.700639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.701584 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.701840 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.701852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.702952 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.702960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.703984 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.703995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.704727 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.704732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.704908 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.705484 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.705500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.707466 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.711455 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.711466 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.711473 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.711990 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.712426 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.712430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.712434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.712613 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.713281 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.713290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.713473 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.713670 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.713966 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3351,"thread_start_us":3204,"threads_started":1}
I20260926 07:27:56.715054 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.696159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.2.147.253:45863)
0926 07:27:56.696335 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.696341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.699986 (+ 3645us) server_negotiation.cc:407] Connection header received
0926 07:27:56.700247 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.700251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.700334 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.700411 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.701988 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.702828 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.704116 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.704590 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.704713 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.707632 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.707637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.707640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.707666 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.711322 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.712123 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.712131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.712137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.712307 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.712742 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.712749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.712760 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.713134 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.713285 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.713815 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.714055 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:56.716980 8777 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.147.253:45863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
member_type: NON_VOTER
last_known_addr {
host: "127.2.147.253"
port: 45863
}
attrs {
promote: true
}
}
dest_uuid: "5beffe5f32114bd492e0e472af2042d5"
cas_config_opid_index: -1
I20260926 07:27:56.718207 8777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } }
I20260926 07:27:56.724466 9335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } }
I20260926 07:27:56.725102 9281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:56.729492 9335 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.147.253:45863
I20260926 07:27:56.729687 9281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.252.keytab.
Entry for principal oryx/127.2.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.252.keytab.
I20260926 07:27:56.730163 9239 master.cc:537] Master@127.2.147.253:45863 shutting down...
I20260926 07:27:56.731329 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:35391
--webserver_interface=127.2.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.252.keytab
--principal=oryx/127.2.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:56.740960 9336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:56.743257 9239 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:56.743646 9239 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:56.743873 9239 tablet_replica.cc:330] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: stopping tablet replica
W20260926 07:27:56.744354 8764 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 07:27:56.745570 9337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } }
I20260926 07:27:56.746627 9337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:56.752256 9239 master.cc:552] Master@127.2.147.253:45863 shutdown complete.
I20260926 07:27:56.752336 9239 master_runner.cc:305] Clearing existing system tablet
I20260926 07:27:56.752864 9239 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:56.752965 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:56.757514 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.756892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41960)
0926 07:27:56.757120 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.757257 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:27:56.758021 8764 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 -> Peer 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253:45863): Couldn't send request to peer 38c78ec5d8ab42c4b782a03ef8f3a497. Status: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:27:56.759639 9239 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:27:56.761068 9343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.761334 9239 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:56.761641 9239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 07:27:56.761754 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "87165de6f8fb4b9ae542a399a516719e"
server_key_iv: "a0194a1e11819c8a6c9a4c61bc715e76"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.765270 9239 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:56.769950 9239 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:27:56.770125 9239 log.cc:1196] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:27:56.770509 9239 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Deleting consensus metadata
I20260926 07:27:56.770778 9239 master_runner.cc:315] Copying system tablet from 127.2.147.254:46117
I20260926 07:27:56.773073 9239 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: tablet copy: Beginning tablet copy session from remote peer at address 127.2.147.254:46117
I20260926 07:27:56.784811 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.773744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.2.147.254:46117)
0926 07:27:56.773941 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:56.773950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:56.774411 (+ 461us) server_negotiation.cc:407] Connection header received
0926 07:27:56.774612 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:56.774618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:56.774700 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:56.774800 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:56.775612 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.776318 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.777500 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.777903 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.778021 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.779733 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:56.779739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:56.779744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:56.779782 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:56.782241 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.782843 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.782851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.782856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.782983 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:56.783387 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:56.783394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:56.783397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:56.783660 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:56.783826 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:56.784249 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 07:27:56.784451 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:27:56.784886 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.773724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41022)
0926 07:27:56.774307 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.774322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:56.774478 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:56.774827 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:56.774831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:56.775260 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:56.775481 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.775490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.776437 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.776444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.777366 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:56.777376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:56.778052 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:56.778058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:56.778226 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:56.778835 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:56.778852 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:56.779540 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:56.782382 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.782388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.782392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.782723 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.783087 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:56.783092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:56.783096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:56.783285 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:56.783770 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:56.783785 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:56.783944 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:27:56.784137 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 07:27:56.784417 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":369,"threads_started":1}
I20260926 07:27:56.785784 8798 tablet_copy_service.cc:140] P 5beffe5f32114bd492e0e472af2042d5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 38c78ec5d8ab42c4b782a03ef8f3a497 ({username='oryx', principal='oryx/127.2.147.253@KRBTEST.COM'} at 127.0.0.1:41022)
I20260926 07:27:56.786034 8798 tablet_copy_service.cc:161] P 5beffe5f32114bd492e0e472af2042d5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 38c78ec5d8ab42c4b782a03ef8f3a497 at {username='oryx', principal='oryx/127.2.147.253@KRBTEST.COM'} at 127.0.0.1:41022: session id = 38c78ec5d8ab42c4b782a03ef8f3a497-00000000000000000000000000000000
I20260926 07:27:56.788935 8798 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:27:56.790696 9239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:27:56.795029 9239 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: tablet copy: Starting download of 0 data blocks...
I20260926 07:27:56.795333 9239 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: tablet copy: Starting download of 1 WAL segments...
I20260926 07:27:56.797355 9239 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:27:56.800248 8798 tablet_copy_service.cc:342] P 5beffe5f32114bd492e0e472af2042d5: Request end of tablet copy session 38c78ec5d8ab42c4b782a03ef8f3a497-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.147.253@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 07:27:56.800407 8798 tablet_copy_service.cc:433] P 5beffe5f32114bd492e0e472af2042d5: ending tablet copy session 38c78ec5d8ab42c4b782a03ef8f3a497-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 38c78ec5d8ab42c4b782a03ef8f3a497
I20260926 07:27:56.810026 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:56.812433 9350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.812915 9351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:56.823431 9353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:56.823640 9239 server_base.cc:1034] running on GCE node
I20260926 07:27:56.823957 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:56.824524 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:56.825733 9239 hybrid_clock.cc:648] HybridClock initialized: now 1790407676825679 us; error 37 us; skew 500 ppm
I20260926 07:27:56.825934 9239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:56 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407676, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:56.827812 9239 init.cc:377] Logged in from keytab as oryx/127.2.147.253@KRBTEST.COM (short username oryx)
I20260926 07:27:56.828943 9239 webserver.cc:460] Webserver started at http://127.2.147.253:38809/ using document root <none> and password file <none>
I20260926 07:27:56.829246 9239 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:56.829334 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:56.831528 9239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:27:56.833099 9360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:56.833498 9239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:27:56.833638 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "87165de6f8fb4b9ae542a399a516719e"
server_key_iv: "a0194a1e11819c8a6c9a4c61bc715e76"
server_key_version: "encryptionkey@0"
I20260926 07:27:56.833799 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:56.839586 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:56.842581 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:56.866153 9239 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.253:45863
I20260926 07:27:56.866173 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.253:45863 every 8 connection(s)
I20260926 07:27:56.867072 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
I20260926 07:27:56.873832 9412 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:27:56.876454 9412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Bootstrap starting.
I20260926 07:27:56.887903 9412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:56.888341 9412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: Bootstrap complete.
I20260926 07:27:56.889184 9412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } }
I20260926 07:27:56.889474 9412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38c78ec5d8ab42c4b782a03ef8f3a497, State: Initialized, Role: LEARNER
I20260926 07:27:56.889694 9412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } }
I20260926 07:27:56.890772 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } }
I20260926 07:27:56.891115 9412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:56.891234 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:27:56.902405 9426 catalog_manager.cc:1259] Loaded cluster ID: 52ba795d1f8b47639a872d597d6e00c3
I20260926 07:27:56.902496 9426 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: loading cluster ID for follower catalog manager: success
I20260926 07:27:56.905740 9426 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: acquiring CA information for follower catalog manager: success
I20260926 07:27:56.908080 9426 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:27:57.117619 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:57.117889 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:57.117987 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:27:57.123427 9234 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
W20260926 07:27:57.126060 9338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:57.126173 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:57.126250 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:57.126346 9338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:57.126416 9338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:57.135896 9338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.252.keytab
--principal=oryx/127.2.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:35391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:57.137120 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:57.138368 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:57.149863 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.149924 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.149938 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:57.152482 9338 server_base.cc:1034] running on GCE node
I20260926 07:27:57.153182 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:57.154453 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:57.155644 9338 hybrid_clock.cc:648] HybridClock initialized: now 1790407677155605 us; error 34 us; skew 500 ppm
I20260926 07:27:57.155957 9338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:57.158910 9338 init.cc:377] Logged in from keytab as oryx/127.2.147.252@KRBTEST.COM (short username oryx)
I20260926 07:27:57.160987 9338 webserver.cc:460] Webserver started at http://127.2.147.252:37767/ using document root <none> and password file <none>
I20260926 07:27:57.161657 9338 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:57.161795 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:57.162101 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:57.164834 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/instance:
uuid: "6427cb5879ab4e1b92cf3afea868a21b"
format_stamp: "Formatted at 2026-09-26 07:27:57 on dist-test-slave-1dwd"
server_key: "757e53bf33fec63d4f1318fd1d570994"
server_key_iv: "04dc84e6eb5f2bbda0a6784e1f677dec"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.165573 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal/instance:
uuid: "6427cb5879ab4e1b92cf3afea868a21b"
format_stamp: "Formatted at 2026-09-26 07:27:57 on dist-test-slave-1dwd"
server_key: "757e53bf33fec63d4f1318fd1d570994"
server_key_iv: "04dc84e6eb5f2bbda0a6784e1f677dec"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.170217 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 07:27:57.173683 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.174932 9338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:57.175102 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "6427cb5879ab4e1b92cf3afea868a21b"
format_stamp: "Formatted at 2026-09-26 07:27:57 on dist-test-slave-1dwd"
server_key: "757e53bf33fec63d4f1318fd1d570994"
server_key_iv: "04dc84e6eb5f2bbda0a6784e1f677dec"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.175328 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:57.189648 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:57.193287 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:57.193506 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:57.211738 9338 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.252:35391
I20260926 07:27:57.211779 9492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.252:35391 every 8 connection(s)
I20260926 07:27:57.212970 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
I20260926 07:27:57.216478 9493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:57.219329 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9338
I20260926 07:27:57.224272 9493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.252@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:57.242287 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.226908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.2.147.254:46117)
0926 07:27:57.227071 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.227075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.227702 (+ 627us) server_negotiation.cc:407] Connection header received
0926 07:27:57.227910 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.227915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.227981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.228079 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.229305 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.230060 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.231670 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.232453 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.232544 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.236179 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.236186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.236190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.236226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.239391 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.240167 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.240172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.240176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.240312 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.240661 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.240666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.240668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.240998 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.241125 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.241719 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.241914 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:27:57.242787 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.221470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.2.147.252:35391)
0926 07:27:57.222639 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.222649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.222686 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:27:57.224225 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.224246 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.224605 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.224909 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.226584 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.228449 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.229888 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.230684 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.230801 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.235388 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.235401 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.235411 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.235446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.238840 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.239523 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.239530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.239538 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.239740 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.240153 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.240157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.240160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.240783 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.241051 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.241385 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.241705 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":980,"thread_start_us":778,"threads_started":1}
I20260926 07:27:57.242815 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.226902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41024)
0926 07:27:57.227574 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.227611 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.227789 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.228126 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.228132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.228882 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.229165 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.229176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.230193 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.230197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.231540 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.231556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.232662 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.232671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.232936 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.233523 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.233551 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.235970 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.239516 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.239531 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.239539 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.240039 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.240426 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.240429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.240431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.240553 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.241129 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.241139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.241387 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.241586 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.241871 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":368,"threads_started":1}
I20260926 07:27:57.244448 9493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.252@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:57.259445 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.245167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41976)
0926 07:27:57.245596 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.245616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.245680 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.246066 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.246068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.246333 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.246481 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.246488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.248073 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.248077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.248927 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.248937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.249517 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.249520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.249633 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.253303 (+ 3670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.253319 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.254977 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.257505 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.257509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.257511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.257967 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.258258 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.258261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.258263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.258377 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.258788 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.258791 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.258890 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.258987 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.259160 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324}
I20260926 07:27:57.259502 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.245212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.2.147.253:45863)
0926 07:27:57.245838 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.245843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.245856 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:27:57.245901 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.245906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.245958 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.246041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.246611 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.247973 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.249032 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.249414 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.249499 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.255133 (+ 5634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.255137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.255140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.255166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.257393 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.258053 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.258057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.258059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.258175 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.258466 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.258469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.258470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.258682 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.258789 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.259065 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.259197 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":411,"threads_started":1}
I20260926 07:27:57.260499 9493 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.252@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
I20260926 07:27:57.271136 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.261085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.0.0.1:51634)
0926 07:27:57.261287 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.261304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.261393 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.261604 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.261607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.261915 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.262043 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.262050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.263084 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.263087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.263930 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.263940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.264532 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.264536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.264672 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.265099 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.265114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.266798 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.269413 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.269416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.269418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.269723 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.270002 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.270005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.270007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.270132 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.270514 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.270517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.270624 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.270723 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.270874 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:27:57.271204 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.261293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.2.147.252:35391)
0926 07:27:57.261398 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.261402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.261416 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:27:57.261468 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.261471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.261515 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.261576 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.262164 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.262986 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.264043 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.264423 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.264517 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.266950 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.266954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.266957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.266981 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.269301 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.269815 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.269818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.269820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.269919 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.270212 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.270215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.270217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.270429 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.270523 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.270808 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.270969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":29}
I20260926 07:27:57.275882 9493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Bootstrap starting.
I20260926 07:27:57.278815 9493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:57.279891 9493 log.cc:824] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:57.282195 9493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: No bootstrap required, opened a new log
I20260926 07:27:57.288882 9493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } }
I20260926 07:27:57.289187 9493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:57.289299 9493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6427cb5879ab4e1b92cf3afea868a21b, State: Initialized, Role: FOLLOWER
I20260926 07:27:57.289813 9493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } }
I20260926 07:27:57.291079 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } } }
I20260926 07:27:57.291435 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:57.291654 9493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:57.297397 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8839
W20260926 07:27:57.300904 9511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:57.301098 9511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:57.304189 9338 master_runner.cc:428] Detected that this master 6427cb5879ab4e1b92cf3afea868a21b is joining an existing cluster
I20260926 07:27:57.304313 9338 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.147.252:35391
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.193@KRBTEST.COM
I20260926 07:27:57.324363 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.298355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.2.147.193:46847)
0926 07:27:57.300390 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.300401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.300432 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:27:57.300500 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.300507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.300810 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.301543 (+ 733us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.302609 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.304290 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.306384 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.307250 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.307377 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.314137 (+ 6760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.314150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.314166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.314210 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.319407 (+ 5197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.320381 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.320389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.320393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.320561 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.321777 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.321782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.321785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.322191 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.322477 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.323689 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.323953 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1689,"threads_started":1}
I20260926 07:27:57.325696 8962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:57.343511 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.317180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.2.147.253:45863)
0926 07:27:57.317337 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.317343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.317497 (+ 154us) server_negotiation.cc:407] Connection header received
0926 07:27:57.317724 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.317731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.317829 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.317901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.319241 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.320277 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.322484 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.323404 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.323525 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.332341 (+ 8816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.332349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.332353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.332391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.336437 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.337379 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.337384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.337388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.337539 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.338008 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.338013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.338016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.338376 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.338481 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.342895 (+ 4414us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.343118 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:27:57.345432 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.316727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41982)
0926 07:27:57.317412 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.317430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.317578 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.318102 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.318106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.318773 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.319062 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.319072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.320502 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.320507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.322144 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.322162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.325398 (+ 3236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.325407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.325691 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.327713 (+ 2022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.327748 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.329127 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.336596 (+ 7469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.336600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.336604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.337243 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.337680 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.337684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.337688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.337882 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.338570 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.338576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.338795 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.344888 (+ 6093us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.345113 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":28,"thread_start_us":4227,"threads_started":1}
I20260926 07:27:57.360188 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.354403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.2.147.253:45863)
0926 07:27:57.354628 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.354636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.355067 (+ 431us) server_negotiation.cc:407] Connection header received
0926 07:27:57.355274 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.355279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.355353 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.355460 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:57.356283 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.356955 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.358435 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.359286 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.359427 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.359639 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.359812 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.359928 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:27:57.360219 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.354227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:41996)
0926 07:27:57.354975 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.354989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.355131 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.355496 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.355504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.355530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:57.356139 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.356148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.357077 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.357083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.358292 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.358303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.359457 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.359463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.359629 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.359689 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.359789 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.359920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":542,"threads_started":1}
I20260926 07:27:57.361254 9387 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 07:27:57.362058 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beffe5f32114bd492e0e472af2042d5. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } }
I20260926 07:27:57.362310 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:27:57.374488 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.315729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41032)
0926 07:27:57.316770 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.316793 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.316940 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.317508 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.317514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.318197 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.318668 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.318680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.320190 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.320195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.322312 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.322327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.325982 (+ 3655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.325991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.326679 (+ 688us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.330271 (+ 3592us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x60c0000347b8. stack: 00007f76c9ef466e 00007f76c9ef4430 00007f76c8f7b5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f76c9eea0fd 00007f76c9ee9d1e 00007f76d0a01664 00007f76d0900810 00007f76d08fd470 00007f76d08f833a 00007f76d097fdca 00007f76d097df68 00007f76d09b182f 00007f76d09b158d
0926 07:27:57.330935 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.330964 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.332137 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.342649 (+ 10512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.342656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.342662 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.371885 (+ 29223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.372394 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.372399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.372406 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.372654 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.373365 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.373374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.373550 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.373670 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.374009 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":197,"spinlock_wait_cycles":5741184,"thread_start_us":725,"threads_started":1}
I20260926 07:27:57.374961 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.321679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.2.147.252:35391)
0926 07:27:57.321826 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.321831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.331741 (+ 9910us) server_negotiation.cc:407] Connection header received
0926 07:27:57.332081 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.332085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.332148 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.332203 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.333541 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.334708 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.337212 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.337935 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.338016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.340797 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.340805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.340813 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.340857 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.344310 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.372391 (+ 28081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.372396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.372404 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.372635 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.373069 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.373074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.373076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.373477 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.373605 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.374063 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.374269 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":3840}
I20260926 07:27:57.375505 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.316355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.2.147.254:46117)
0926 07:27:57.316536 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.316544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.316874 (+ 330us) server_negotiation.cc:407] Connection header received
0926 07:27:57.317133 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.317141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.317228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.317301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.318855 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.320003 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.323767 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.324629 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.324745 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.339073 (+ 14328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.339082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.339090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.339131 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.342478 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.372057 (+ 29579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.372062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.372066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.372251 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.372792 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.372796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.372800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.373162 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.373307 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.374899 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.375148 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:27:57.376397 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.321351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.0.0.1:51642)
0926 07:27:57.331656 (+ 10305us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.331673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.331957 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.332373 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.332377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.333040 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.333401 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.333412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.334975 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.334980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.337076 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.337088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.338313 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.338320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.338477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.339146 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.339184 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.340595 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.344488 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.344492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.344494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.372267 (+ 27773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.372799 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.372804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.372808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.372959 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.373668 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.373672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.373827 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.375390 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.375975 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10174,"thread_start_us":10406,"threads_started":1}
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.254@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
I20260926 07:27:57.393770 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.380569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.0.0.1:51652)
0926 07:27:57.380792 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.380812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.380916 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.381268 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.381275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.381754 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.381909 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.381917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.382905 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.382912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.383808 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.383818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.384915 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.384921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.385078 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.385730 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.385758 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.388294 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.391141 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.391147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.391153 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.391590 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.391963 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.391968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.391971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.392084 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.392719 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.392726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.392907 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.393087 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.393366 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 07:27:57.393752 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.380642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.2.147.252:35391)
0926 07:27:57.380789 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.380795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.380861 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:27:57.381056 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.381061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.381146 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.381251 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.382048 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.382786 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.383928 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.384614 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.384726 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.388461 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.388467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.388471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.388507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.391021 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.391708 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.391716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.391719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.391833 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.392219 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.392224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.392225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.392585 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.392799 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.393170 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.393366 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:27:57.395349 8777 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.147.252:35391: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b"
member_type: NON_VOTER
last_known_addr {
host: "127.2.147.252"
port: 35391
}
attrs {
promote: true
}
}
dest_uuid: "5beffe5f32114bd492e0e472af2042d5"
cas_config_opid_index: 5
I20260926 07:27:57.396193 8777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } }
I20260926 07:27:57.397959 9522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.398741 9468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:57.402254 9522 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.147.252:35391
I20260926 07:27:57.402745 9468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:27:57.402984 9387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:27:57.404237 9338 master.cc:537] Master@127.2.147.252:35391 shutting down...
I20260926 07:27:57.415937 9526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:57.417529 9526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260926 07:27:57.421087 8764 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 -> Peer 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252:35391): Couldn't send request to peer 6427cb5879ab4e1b92cf3afea868a21b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:27:57.421825 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.422164 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:57.422482 9387 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.428360 9338 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:57.428418 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: NON_VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: true } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.428671 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:27:57.428877 9338 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:57.429215 9338 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: stopping tablet replica
I20260926 07:27:57.430178 9524 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: attempting to promote NON_VOTER 38c78ec5d8ab42c4b782a03ef8f3a497 to VOTER
I20260926 07:27:57.430814 9524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } }
I20260926 07:27:57.433413 9387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 LEARNER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 07:27:57.434175 9522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 07:27:57.434830 8764 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 -> Peer 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252:35391): Couldn't send request to peer 6427cb5879ab4e1b92cf3afea868a21b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:27:57.436914 9525 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.437435 9386 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.441327 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.441576 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:57.442644 9338 master.cc:552] Master@127.2.147.252:35391 shutdown complete.
I20260926 07:27:57.442737 9338 master_runner.cc:305] Clearing existing system tablet
I20260926 07:27:57.443070 9338 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:57.443186 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:57.445587 9522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.445544 9524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.445911 9524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:57.445832 9522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:57.446099 9338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:57.447599 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.447890 9338 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:57.448172 9338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:27:57.448277 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "6427cb5879ab4e1b92cf3afea868a21b"
format_stamp: "Formatted at 2026-09-26 07:27:57 on dist-test-slave-1dwd"
server_key: "757e53bf33fec63d4f1318fd1d570994"
server_key_iv: "04dc84e6eb5f2bbda0a6784e1f677dec"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.451328 9338 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:57.456205 9338 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:27:57.456373 9338 log.cc:1196] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:27:57.456773 9338 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Deleting consensus metadata
I20260926 07:27:57.457036 9338 master_runner.cc:315] Copying system tablet from 127.2.147.254:46117
I20260926 07:27:57.459342 9338 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: tablet copy: Beginning tablet copy session from remote peer at address 127.2.147.254:46117
I20260926 07:27:57.473977 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.459994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.2.147.254:46117)
0926 07:27:57.460249 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.460258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.463686 (+ 3428us) server_negotiation.cc:407] Connection header received
0926 07:27:57.463889 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.463894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.463951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.464045 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.464851 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.465564 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.466737 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.467129 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.467289 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.468987 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:57.468997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:57.469000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:57.469031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:57.471542 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.472158 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.472166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.472169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.472295 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:57.472638 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:57.472645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:57.472648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:57.472898 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:57.473039 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:57.473405 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:27:57.473624 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 07:27:57.474014 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.459974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:41046)
0926 07:27:57.463588 (+ 3614us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.463603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.463754 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.464066 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.464071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.464518 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.464697 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.464706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.465698 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.465703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.466593 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.466603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.467295 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.467300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.467473 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.468107 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.468123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:57.468807 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:57.471665 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.471672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.471676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.472042 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.472396 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:57.472402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:57.472406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:57.472541 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:57.473011 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:57.473016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:57.473168 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 07:27:57.473304 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 07:27:57.473586 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3484,"thread_start_us":3388,"threads_started":1}
I20260926 07:27:57.474807 8798 tablet_copy_service.cc:140] P 5beffe5f32114bd492e0e472af2042d5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6427cb5879ab4e1b92cf3afea868a21b ({username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:41046)
I20260926 07:27:57.474973 8798 tablet_copy_service.cc:161] P 5beffe5f32114bd492e0e472af2042d5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6427cb5879ab4e1b92cf3afea868a21b at {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:41046: session id = 6427cb5879ab4e1b92cf3afea868a21b-00000000000000000000000000000000
I20260926 07:27:57.477901 8798 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:27:57.479452 9338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:27:57.480793 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8839
I20260926 07:27:57.484912 9338 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: tablet copy: Starting download of 0 data blocks...
I20260926 07:27:57.485163 9338 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: tablet copy: Starting download of 1 WAL segments...
I20260926 07:27:57.487437 9338 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:27:57.491457 8798 tablet_copy_service.cc:342] P 5beffe5f32114bd492e0e472af2042d5: Request end of tablet copy session 6427cb5879ab4e1b92cf3afea868a21b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:41046
I20260926 07:27:57.491582 8798 tablet_copy_service.cc:433] P 5beffe5f32114bd492e0e472af2042d5: ending tablet copy session 6427cb5879ab4e1b92cf3afea868a21b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6427cb5879ab4e1b92cf3afea868a21b
I20260926 07:27:57.502166 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:46847
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:46117
--webserver_port=39461
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:57.503083 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:57.505635 9538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.508944 9539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.528822 9542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:57.529497 9338 server_base.cc:1034] running on GCE node
I20260926 07:27:57.529846 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:57.530478 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:57.531658 9338 hybrid_clock.cc:648] HybridClock initialized: now 1790407677531618 us; error 33 us; skew 500 ppm
I20260926 07:27:57.531886 9338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:57.533962 9338 init.cc:377] Logged in from keytab as oryx/127.2.147.252@KRBTEST.COM (short username oryx)
I20260926 07:27:57.540195 9338 webserver.cc:460] Webserver started at http://127.2.147.252:41395/ using document root <none> and password file <none>
I20260926 07:27:57.540515 9338 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:57.540608 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:57.543640 9338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:27:57.545476 9549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.545914 9338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:27:57.546072 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "6427cb5879ab4e1b92cf3afea868a21b"
format_stamp: "Formatted at 2026-09-26 07:27:57 on dist-test-slave-1dwd"
server_key: "757e53bf33fec63d4f1318fd1d570994"
server_key_iv: "04dc84e6eb5f2bbda0a6784e1f677dec"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.546319 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:57.568531 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:57.572450 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:57.599450 9338 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.252:35391
I20260926 07:27:57.599538 9600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.252:35391 every 8 connection(s)
I20260926 07:27:57.600548 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
I20260926 07:27:57.605854 9601 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:27:57.609700 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Bootstrap starting.
I20260926 07:27:57.627488 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:57.627956 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Bootstrap complete.
I20260926 07:27:57.628942 9601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } }
I20260926 07:27:57.629221 9601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6427cb5879ab4e1b92cf3afea868a21b, State: Initialized, Role: LEARNER
I20260926 07:27:57.629468 9601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } }
I20260926 07:27:57.630481 9601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:57.630224 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:57.630779 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: LEARNER
I20260926 07:27:57.639626 9615 catalog_manager.cc:1259] Loaded cluster ID: 52ba795d1f8b47639a872d597d6e00c3
I20260926 07:27:57.639724 9615 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: loading cluster ID for follower catalog manager: success
I20260926 07:27:57.643159 9615 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: acquiring CA information for follower catalog manager: success
I20260926 07:27:57.645473 9615 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:27:57.857633 9540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:57.857951 9540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:57.858100 9540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:57.865429 9540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:57.865554 9540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:57.865734 9540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:27:57.880249 9540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=39461
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:57.881496 9540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:57.882750 9540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:57.896226 9620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.896226 9623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:57.896273 9621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:57.896941 9540 server_base.cc:1034] running on GCE node
I20260926 07:27:57.897513 9540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:57.898415 9540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:57.899595 9540 hybrid_clock.cc:648] HybridClock initialized: now 1790407677899560 us; error 36 us; skew 500 ppm
I20260926 07:27:57.899884 9540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:57.902760 9540 init.cc:377] Logged in from keytab as oryx/127.2.147.193@KRBTEST.COM (short username oryx)
I20260926 07:27:57.904090 9540 webserver.cc:460] Webserver started at http://127.2.147.193:39461/ using document root <none> and password file <none>
I20260926 07:27:57.904702 9540 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:57.904850 9540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:57.909910 9540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:27:57.912894 9630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.914078 9540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 07:27:57.914233 9540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b98a232158b547f38305ad1fb7546748"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "ddb7c2a620bbada49e1fe69e4d8898cb"
server_key_iv: "44f0ebd4ede6150a9082fef879bcfb41"
server_key_version: "encryptionkey@0"
I20260926 07:27:57.915122 9540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:57.928776 9540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:57.932873 9540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:57.933126 9540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:57.934059 9540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:57.935518 9540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:57.935622 9540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.935721 9540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:57.935796 9540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:57.969709 9540 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:46847
I20260926 07:27:57.970201 9742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:46847 every 8 connection(s)
I20260926 07:27:57.971398 9540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:57.979946 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9540
I20260926 07:27:57.980320 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8973
Sep 26 07:27:57 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407677, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:58.011747 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.974133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:56869 (local address 127.2.147.252:35391)
0926 07:27:57.977682 (+ 3549us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.977689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.978300 (+ 611us) server_negotiation.cc:407] Connection header received
0926 07:27:57.984693 (+ 6393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.984701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.984799 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.984915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.986642 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.988650 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.995544 (+ 6894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.996401 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.996544 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.004202 (+ 7658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.004209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.004214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.004258 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.008036 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.008944 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.008951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.008956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.009092 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.009819 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.009826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.009830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.010154 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.010258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.011111 (+ 853us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.011336 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3414,"thread_start_us":3265,"threads_started":1}
I20260926 07:27:58.012844 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.974643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.2.147.193:56869)
0926 07:27:57.978108 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.978165 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.980658 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.985182 (+ 4524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.985189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.986086 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.986432 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.986447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.988863 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.988871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.995355 (+ 6484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.995370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.996905 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.996913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.997766 (+ 853us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:57.998726 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:57.998760 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.003475 (+ 4715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.008198 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.008207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.008224 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.008819 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.009357 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.009363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.009367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.009585 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.010375 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.010383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.010765 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.011027 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.011526 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3106,"mutex_wait_us":4,"thread_start_us":2875,"threads_started":1}
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.194@KRBTEST.COM
I20260926 07:27:58.014575 9743 heartbeater.cc:347] Connected to a master server at 127.2.147.252:35391
I20260926 07:27:58.014873 9743 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.015480 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.982365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.2.147.193:40669)
0926 07:27:57.991886 (+ 9521us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.991908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.992037 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.996324 (+ 4287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.996331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.997057 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.997528 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.997543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.999238 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.999246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.002489 (+ 3243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.002502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.004021 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.004030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.004224 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.004736 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:58.004761 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.008108 (+ 3347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.012644 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.012650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.012653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.013200 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.013623 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.013628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.013631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.013804 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.014596 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.014600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.014755 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.014893 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.015123 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9063,"mutex_wait_us":25,"thread_start_us":8923,"threads_started":1}
I20260926 07:27:58.015734 9743 heartbeater.cc:511] Master 127.2.147.252:35391 requested a full tablet report, sending...
I20260926 07:27:58.016042 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.978998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:40669 (local address 127.2.147.253:45863)
0926 07:27:57.995815 (+ 16817us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.995823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.995854 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:27:57.995937 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.995945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.996047 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.997268 (+ 1221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.998374 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.999086 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.002712 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.003385 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.003495 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.008340 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.008366 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.008370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.008402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.012514 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.013328 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.013334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.013337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.013480 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.014112 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.014117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.014123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.014468 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.014595 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.015068 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.015303 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16665,"thread_start_us":17149,"threads_started":1}
I20260926 07:27:58.016810 9745 heartbeater.cc:347] Connected to a master server at 127.2.147.253:45863
I20260926 07:27:58.016927 9745 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.017328 9745 heartbeater.cc:511] Master 127.2.147.253:45863 requested a full tablet report, sending...
I20260926 07:27:58.018157 9566 ts_manager.cc:194] Registered new tserver with Master: b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:58.018565 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.974643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:39513)
0926 07:27:57.981345 (+ 6702us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.981363 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:57.981552 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:57.982411 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:57.982425 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:57.983481 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:57.984452 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.984481 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.992949 (+ 8468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.992957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:57.995022 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:57.995040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.999856 (+ 4816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.999864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.001419 (+ 1555us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.002164 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:58.002186 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.005554 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.013427 (+ 7873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.013432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.013435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.013956 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.016523 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.016528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.016530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.016674 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.017635 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.017639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.017787 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.017915 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.018191 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6546,"mutex_wait_us":27,"spinlock_wait_cycles":68992,"thread_start_us":6309,"threads_started":1}
I20260926 07:27:58.018953 9377 ts_manager.cc:194] Registered new tserver with Master: b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:58.019841 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.974103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:39513 (local address 127.2.147.254:46117)
0926 07:27:57.974524 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.974533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:57.981445 (+ 6912us) server_negotiation.cc:407] Connection header received
0926 07:27:57.981707 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:57.981715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:57.981809 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:57.981887 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:57.991338 (+ 9451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.992686 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.997544 (+ 4858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:57.998090 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:57.998163 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.005768 (+ 7605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.005775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.005780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.005819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.009932 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.014276 (+ 4344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.014281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.014284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.014410 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.016769 (+ 2359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.016773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.016777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.017097 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.017286 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.019255 (+ 1969us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.019475 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260926 07:27:58.020536 9744 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:58.020659 9744 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.021054 9744 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:58.021361 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.981643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.2.147.194:33103)
0926 07:27:57.997842 (+ 16199us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:57.997851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.000044 (+ 2193us) server_negotiation.cc:407] Connection header received
0926 07:27:58.000135 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.000142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.000442 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.000608 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:58.002837 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.004724 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.007498 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.008387 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.008505 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.014761 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.014770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.014781 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.014808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.018301 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.019226 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.019234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.019236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.019359 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.019821 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.019825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.019829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.020219 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.020465 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.020806 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.021028 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15801,"thread_start_us":15841,"threads_started":1}
I20260926 07:27:58.021935 8777 ts_manager.cc:194] Re-registered known tserver with Master: b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:58.022339 9096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:58.023008 8777 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:39513
I20260926 07:27:58.032948 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.026806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.2.147.252:35391)
0926 07:27:58.027011 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:58.027017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.027349 (+ 332us) server_negotiation.cc:407] Connection header received
0926 07:27:58.027588 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.027594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.027692 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.027773 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:58.028349 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.028981 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.031151 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.032125 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.032241 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.032358 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.032657 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.032779 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:27:58.033116 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.026688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.0.0.1:51658)
0926 07:27:58.027256 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.027277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:58.027442 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:58.027961 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:58.027967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:58.027995 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:58.028186 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.028199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.029111 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.029116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.031021 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.031034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.032300 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.032308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.032499 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.032538 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.032644 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.032785 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260926 07:27:58.034020 9576 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 07:27:58.034616 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beffe5f32114bd492e0e472af2042d5. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:58.034881 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: LEARNER
I20260926 07:27:58.106650 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8973
I20260926 07:27:58.123704 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:33103
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:46117
--webserver_port=38321
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:58.434340 9757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:58.434646 9757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:58.434819 9757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:58.441886 9757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:58.441977 9757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:58.442086 9757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:27:58.451540 9757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:33103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=38321
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:58.452786 9757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:58.454123 9757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:58.466949 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:58.467017 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:58.467017 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:58.467859 9757 server_base.cc:1034] running on GCE node
I20260926 07:27:58.468446 9757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:58.469292 9757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:58.470538 9757 hybrid_clock.cc:648] HybridClock initialized: now 1790407678470496 us; error 48 us; skew 500 ppm
I20260926 07:27:58.470858 9757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407678, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:58.473943 9757 init.cc:377] Logged in from keytab as oryx/127.2.147.194@KRBTEST.COM (short username oryx)
I20260926 07:27:58.475373 9757 webserver.cc:460] Webserver started at http://127.2.147.194:38321/ using document root <none> and password file <none>
I20260926 07:27:58.476018 9757 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:58.476177 9757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:58.481453 9757 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:27:58.484231 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:58.485671 9757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:58.485834 9757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "425211964e7a4614a119f70d6f77fbc3"
format_stamp: "Formatted at 2026-09-26 07:27:55 on dist-test-slave-1dwd"
server_key: "aba93cbfb132e905f4282fa3c3e962b2"
server_key_iv: "a6b4aa930569a6ae884c8a9ec63e1c6c"
server_key_version: "encryptionkey@0"
I20260926 07:27:58.486759 9757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:58.500116 9757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:58.504194 9757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:58.504451 9757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:58.505353 9757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:58.506589 9757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:58.506693 9757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:58.506810 9757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:58.506899 9757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:58.538837 9757 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:33103
I20260926 07:27:58.538926 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:33103 every 8 connection(s)
I20260926 07:27:58.540191 9757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:58.542322 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9757
I20260926 07:27:58.542616 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9106
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.195@KRBTEST.COM
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407678, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:27:58.570799 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.544064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.2.147.195:36619)
0926 07:27:58.549784 (+ 5720us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:58.549792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.549853 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:27:58.549914 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.549920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.550085 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.550269 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:58.552230 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.553847 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.555792 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.556594 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.556686 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.562802 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.562814 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.562829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.562868 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.566798 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.568351 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.568359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.568363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.568514 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.568943 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.568948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.568951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.569323 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.569570 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.570152 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.570405 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5572,"thread_start_us":5434,"threads_started":1}
I20260926 07:27:58.573483 9229 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407678, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
I20260926 07:27:58.580636 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.543545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:35899 (local address 127.2.147.254:46117)
0926 07:27:58.554296 (+ 10751us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:58.554304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.555937 (+ 1633us) server_negotiation.cc:407] Connection header received
0926 07:27:58.556000 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.556005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.556087 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.556193 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:58.558819 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.560260 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.563128 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.564000 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.564111 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.569110 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.569119 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.569127 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.569177 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.573700 (+ 4523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.574920 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.574928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.574934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.575111 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.576119 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.576127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.576133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.576553 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.576698 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.579446 (+ 2748us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.579718 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10610,"thread_start_us":10523,"threads_started":1}
I20260926 07:27:58.584658 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.544584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:35899)
0926 07:27:58.555686 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.555705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:58.555828 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:58.556553 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:58.556564 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:58.557867 (+ 1303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:58.558577 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.558609 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.560490 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.560498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.562934 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.562952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.564476 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.564485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.564820 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.565497 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:58.565539 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.568839 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.573933 (+ 5094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.573945 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.573964 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.574748 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.575614 (+ 866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.575622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.575628 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.575930 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.576802 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.576813 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.577315 (+ 502us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.577715 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.578340 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10980,"mutex_wait_us":37,"thread_start_us":7202,"threads_started":1}
I20260926 07:27:58.586198 9886 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:58.586542 9886 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.587416 9886 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:58.587831 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.543776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55579 (local address 127.2.147.252:35391)
0926 07:27:58.553905 (+ 10129us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:58.553913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.553939 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:27:58.555967 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.555973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.556050 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.556147 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:58.569935 (+ 13788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.570979 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.573331 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.574119 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.574260 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.579021 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.579030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.579037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.579088 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.583461 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.584439 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.584448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.584454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.584631 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.585477 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.585484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.585489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.585908 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.586071 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.586876 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.587149 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10006,"thread_start_us":9937,"threads_started":1}
I20260926 07:27:58.589098 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.543533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.2.147.194:55579)
0926 07:27:58.552268 (+ 8735us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.552316 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:58.555594 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:58.556550 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:58.556561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:58.557476 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:58.567326 (+ 9850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.567340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.571260 (+ 3920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.571265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.573130 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.573144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.574622 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.574631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.574865 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.575553 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:58.575582 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.578769 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.583650 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.583658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.583665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.584263 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.585124 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.585132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.585137 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.585329 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.586192 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.586199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.586498 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.588294 (+ 1796us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.588684 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8362,"mutex_wait_us":29,"thread_start_us":11783,"threads_started":1}
I20260926 07:27:58.589079 8777 ts_manager.cc:194] Re-registered known tserver with Master: 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103)
I20260926 07:27:58.589931 9885 heartbeater.cc:347] Connected to a master server at 127.2.147.252:35391
I20260926 07:27:58.590052 9885 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.590220 8777 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:35899
I20260926 07:27:58.590427 9885 heartbeater.cc:511] Master 127.2.147.252:35391 requested a full tablet report, sending...
I20260926 07:27:58.592097 9566 ts_manager.cc:194] Registered new tserver with Master: 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103)
Sep 26 07:27:58 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407678, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:58.601074 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.562123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.2.147.194:36111)
0926 07:27:58.567653 (+ 5530us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.567675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:58.567800 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:58.585451 (+ 17651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:58.585456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:58.586032 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:58.586349 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.586361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.587775 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.587780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.589117 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:58.589128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.591649 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.591659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:58.591955 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.592624 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:58.592644 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:58.595078 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:58.598247 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.598254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.598259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.598744 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.599126 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:58.599130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:58.599132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:58.599342 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:58.600104 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:58.600112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:58.600253 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 07:27:58.600386 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:27:58.600697 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5393,"mutex_wait_us":24,"thread_start_us":5282,"threads_started":1}
I20260926 07:27:58.601300 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.556029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:36111 (local address 127.2.147.253:45863)
0926 07:27:58.584672 (+ 28643us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:58.584681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:58.584701 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:27:58.584760 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:58.584765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:58.585215 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:58.585326 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:58.586524 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.587616 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.589306 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:58.590743 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:58.590885 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:58.595281 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:58.595286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:58.595289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:58.595316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:58.598115 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.598867 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.598872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.598875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.599008 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:58.599529 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:58.599535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:58.599539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:58.599957 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:58.600147 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:58.600568 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 07:27:58.600816 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28506,"thread_start_us":28405,"threads_started":1}
I20260926 07:27:58.602029 9887 heartbeater.cc:347] Connected to a master server at 127.2.147.253:45863
I20260926 07:27:58.602134 9887 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:58.602507 9887 heartbeater.cc:511] Master 127.2.147.253:45863 requested a full tablet report, sending...
I20260926 07:27:58.603549 9377 ts_manager.cc:194] Registered new tserver with Master: 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103)
I20260926 07:27:58.617780 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: NON_VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: true } } }
I20260926 07:27:58.618093 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:58.619853 9898 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: attempting to promote NON_VOTER 6427cb5879ab4e1b92cf3afea868a21b to VOTER
I20260926 07:27:58.620466 9898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } }
I20260926 07:27:58.622386 9387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 07:27:58.622933 9576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 LEARNER]: Refusing update from remote peer 5beffe5f32114bd492e0e472af2042d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 07:27:58.622959 9898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 07:27:58.623497 9899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 07:27:58.625697 9898 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.626283 9387 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.629626 9898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.629566 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.629904 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:58.629993 9898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:58.631695 9904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.632293 9904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:58.634009 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.634265 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:58.634624 9576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.637804 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beffe5f32114bd492e0e472af2042d5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:27:58.638713 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:58.681035 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9106
I20260926 07:27:58.697295 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:36619
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:46117
--webserver_port=46883
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:59.013607 9906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:59.014101 9906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:59.014276 9906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:59.022017 9906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:59.022138 9906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:59.022327 9906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:27:59.025518 9744 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
I20260926 07:27:59.031616 9906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:36619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=46883
--tserver_master_addrs=127.2.147.254:46117,127.2.147.253:45863,127.2.147.252:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:27:59.032805 9906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:59.034092 9906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:59.046351 9912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:59.046396 9911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:59.046440 9914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:59.047008 9906 server_base.cc:1034] running on GCE node
I20260926 07:27:59.047611 9906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:59.048480 9906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:59.049687 9906 hybrid_clock.cc:648] HybridClock initialized: now 1790407679049587 us; error 75 us; skew 500 ppm
I20260926 07:27:59.049961 9906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:27:59 dist-test-slave-1dwd krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:59.052886 9906 init.cc:377] Logged in from keytab as oryx/127.2.147.195@KRBTEST.COM (short username oryx)
I20260926 07:27:59.054210 9906 webserver.cc:460] Webserver started at http://127.2.147.195:46883/ using document root <none> and password file <none>
I20260926 07:27:59.054885 9906 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:59.055025 9906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:59.059859 9906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:59.062438 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:59.063789 9906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:27:59.063956 9906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "6576e30eaac644bfbc4a2a612b70b887"
format_stamp: "Formatted at 2026-09-26 07:27:56 on dist-test-slave-1dwd"
server_key: "81489bac324c620f49496c2578a92cb8"
server_key_iv: "ee0aa6456709157e1a365ee2fc89a029"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.064837 9906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:59.093293 9906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:59.097335 9906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:59.097596 9906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:59.098543 9906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:59.100142 9906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:59.100303 9906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:59.100426 9906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:59.100517 9906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:59.135473 9906 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:36619
I20260926 07:27:59.135576 10033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:36619 every 8 connection(s)
I20260926 07:27:59.136857 9906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:59.145360 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 9906
Sep 26 07:27:59 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
Sep 26 07:27:59 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.252@KRBTEST.COM
Sep 26 07:27:59 dist-test-slave-1dwd krb5kdc[8733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.253@KRBTEST.COM
I20260926 07:27:59.174084 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.139558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43355 (local address 127.2.147.254:46117)
0926 07:27:59.152286 (+ 12728us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.152295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.153940 (+ 1645us) server_negotiation.cc:407] Connection header received
0926 07:27:59.154194 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.154199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.154270 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.154345 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.156238 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.157268 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.159133 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.159968 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.160060 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.164879 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.164889 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.164896 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.164945 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.169383 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.170831 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.170840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.170846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.171032 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.171566 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.171574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.171580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.172001 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.172178 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.173506 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.173718 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12570,"thread_start_us":13440,"threads_started":1}
I20260926 07:27:59.174835 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.140680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:43355)
0926 07:27:59.153826 (+ 13146us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.153859 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.154048 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.154535 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.154540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.155164 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:59.156090 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.156099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.157459 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.157464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.159003 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.159017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.160320 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.160325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.160470 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.161522 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:59.161541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:59.164661 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:59.170101 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.170108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.170111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.170716 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.171158 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.171165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.171170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.171431 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.172171 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:59.172180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:59.172600 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.172944 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.173447 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13022,"mutex_wait_us":52,"spinlock_wait_cycles":16768,"thread_start_us":14667,"threads_started":1}
I20260926 07:27:59.175369 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.141888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:47775 (local address 127.2.147.253:45863)
0926 07:27:59.150397 (+ 8509us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.150406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.155744 (+ 5338us) server_negotiation.cc:407] Connection header received
0926 07:27:59.155860 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.155885 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.156002 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.156137 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.157924 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.159348 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.162517 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.163397 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.163514 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.169072 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.169080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.169085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.169125 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.172338 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.173132 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.173137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.173141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.173280 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.173693 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.173698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.173701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.174025 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.174135 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.174764 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.174984 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8366,"thread_start_us":8266,"threads_started":1}
I20260926 07:27:59.175921 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.141748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.2.147.195:47775)
0926 07:27:59.155622 (+ 13874us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.155640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.155751 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.156349 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.156357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.157117 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:59.157692 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.157709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.159578 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.159585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.162322 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.162341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.164545 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.164552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.164740 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.165710 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:59.165730 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:59.168422 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:59.172493 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.172497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.172501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.173015 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.173421 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.173425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.173428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.173574 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.174220 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:59.174224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:59.174401 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.175280 (+ 879us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.175553 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13746,"thread_start_us":11799,"threads_started":1}
I20260926 07:27:59.176723 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.140644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:46005 (local address 127.2.147.252:35391)
0926 07:27:59.152632 (+ 11988us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.152636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.152660 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:27:59.152709 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.152712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.152768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.152901 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.155172 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.156387 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.158949 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.159566 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.159657 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.165656 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.165665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.165672 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.165709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.169652 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.171232 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.171239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.171245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.171421 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.172765 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.172774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.172780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.173202 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.173341 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.175955 (+ 2614us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.176386 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11882,"thread_start_us":11791,"threads_started":1}
I20260926 07:27:59.176954 10036 heartbeater.cc:347] Connected to a master server at 127.2.147.253:45863
I20260926 07:27:59.177213 10036 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:59.177256 10035 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46117
I20260926 07:27:59.177413 10035 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:59.178109 10035 heartbeater.cc:511] Master 127.2.147.254:46117 requested a full tablet report, sending...
I20260926 07:27:59.178408 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.140115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.2.147.195:46005)
0926 07:27:59.140901 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.140957 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.143777 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.153145 (+ 9368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.153160 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.154048 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:59.154944 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.154969 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.156581 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.156589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.158778 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.158793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.160091 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.160098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.160408 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.161419 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:59.161458 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:59.165484 (+ 4026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:59.169802 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.169814 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.169835 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.170648 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.172284 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:59.172292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:59.172298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:59.172542 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:59.174615 (+ 2073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:59.174619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:59.174778 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.177410 (+ 2632us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.177870 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":23,"thread_start_us":655,"threads_started":1}
I20260926 07:27:59.179491 10034 heartbeater.cc:347] Connected to a master server at 127.2.147.252:35391
I20260926 07:27:59.179528 10036 heartbeater.cc:511] Master 127.2.147.253:45863 requested a full tablet report, sending...
I20260926 07:27:59.179630 10034 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:59.179629 8777 ts_manager.cc:194] Re-registered known tserver with Master: 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:27:59.180109 10034 heartbeater.cc:511] Master 127.2.147.252:35391 requested a full tablet report, sending...
I20260926 07:27:59.180996 8777 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:43355
I20260926 07:27:59.181429 9377 ts_manager.cc:194] Registered new tserver with Master: 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:27:59.182821 9566 ts_manager.cc:194] Registered new tserver with Master: 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:27:59.202591 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.176981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.2.147.253:45863)
0926 07:27:59.182566 (+ 5585us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.182572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.186903 (+ 4331us) server_negotiation.cc:407] Connection header received
0926 07:27:59.187076 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.187080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.187145 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.187280 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.189065 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.190108 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.191943 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.192600 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.192763 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.196762 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.196770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.196776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.196814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.199570 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.200549 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.200555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.200558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.200698 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.201104 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.201108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.201112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.201433 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.201712 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.202068 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.202295 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5470}
I20260926 07:27:59.204452 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.179014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.2.147.254:46117)
0926 07:27:59.182038 (+ 3024us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.182044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.185299 (+ 3255us) server_negotiation.cc:407] Connection header received
0926 07:27:59.185491 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.185496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.185575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.185674 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.188688 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.189729 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.192054 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.192709 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.192826 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.196118 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.196124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.196128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.196163 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.199544 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.200576 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.200583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.200589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.200716 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.201841 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.201846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.201850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.202195 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.202405 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.203485 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.203671 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2916}
I20260926 07:27:59.206261 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.185929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.2.147.252:35391)
0926 07:27:59.186079 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.186083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.189986 (+ 3903us) server_negotiation.cc:407] Connection header received
0926 07:27:59.190173 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.190179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.190286 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.190374 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.192473 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.193402 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.196026 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.196680 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.196812 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.200021 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.200031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.200036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.200063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.202998 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.203869 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.203874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.203877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.204028 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.204849 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.204853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.204856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.205198 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.205415 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.205730 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.205938 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:27:59.208585 8777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:59.215263 8777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:59.242076 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.232184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36619 (local address 127.0.0.1:37240)
0926 07:27:59.232823 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.232849 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.233441 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.234309 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.234313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.234331 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:59.234786 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.234804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.237002 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.237027 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.239572 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.239585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.241369 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.241375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.241533 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.241565 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.241654 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.241761 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":300,"threads_started":1}
I20260926 07:27:59.242182 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.232482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.2.147.195:36619)
0926 07:27:59.233714 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.233724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.233770 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:27:59.233872 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.233880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.234139 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.234336 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:59.235444 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.236865 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.239724 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.241194 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.241312 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.241500 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.241713 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.241854 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":653,"threads_started":1}
I20260926 07:27:59.243122 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.233301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.2.147.193:46847)
0926 07:27:59.233883 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.233894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.234230 (+ 336us) server_negotiation.cc:407] Connection header received
0926 07:27:59.235657 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.235666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.235943 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.236121 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:59.236705 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.238081 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.239897 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.241185 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.241314 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.241504 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.242693 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.242842 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":507,"threads_started":1}
I20260926 07:27:59.243237 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.233688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:46847 (local address 127.0.0.1:56758)
0926 07:27:59.234131 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.234144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.235485 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.236122 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.236130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.236164 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:59.236530 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.236543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.238205 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.238210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.239763 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.239776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.242337 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.242342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.242517 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.242545 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.242636 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.242766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"spinlock_wait_cycles":384,"thread_start_us":315,"threads_started":1}
I20260926 07:27:59.245700 9969 tablet_service.cc:1467] Processing CreateTablet for tablet 70b5390527ce437c98b3735afa7832bc (DEFAULT_TABLE table=test-table [id=6842f70717904b9496b0711808f13ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:59.246497 9678 tablet_service.cc:1467] Processing CreateTablet for tablet 70b5390527ce437c98b3735afa7832bc (DEFAULT_TABLE table=test-table [id=6842f70717904b9496b0711808f13ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:59.247064 9969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b5390527ce437c98b3735afa7832bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:59.247761 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b5390527ce437c98b3735afa7832bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:59.248827 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.234541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33103 (local address 127.0.0.1:41160)
0926 07:27:59.237466 (+ 2925us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.237482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.237588 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.240075 (+ 2487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.240079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.240103 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:59.240473 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.240487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.242778 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.242782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.245432 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.245446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.248102 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.248108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.248276 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.248307 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.248391 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.248498 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2814,"thread_start_us":782,"threads_started":1}
I20260926 07:27:59.249496 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.234530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.2.147.194:33103)
0926 07:27:59.239572 (+ 5042us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.239580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.239607 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:27:59.239699 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.239708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.239916 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.240094 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:59.240664 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.242626 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.245595 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.247636 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.247768 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.247962 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.249085 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.249236 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4908,"thread_start_us":4807,"threads_started":1}
I20260926 07:27:59.252465 9820 tablet_service.cc:1467] Processing CreateTablet for tablet 70b5390527ce437c98b3735afa7832bc (DEFAULT_TABLE table=test-table [id=6842f70717904b9496b0711808f13ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:59.253665 9820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b5390527ce437c98b3735afa7832bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:59.258471 10065 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Bootstrap starting.
I20260926 07:27:59.262120 10065 tablet_bootstrap.cc:654] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:59.262449 10066 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Bootstrap starting.
I20260926 07:27:59.263872 10065 log.cc:824] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:59.266024 10066 tablet_bootstrap.cc:654] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:59.266851 10065 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: No bootstrap required, opened a new log
I20260926 07:27:59.267130 10065 ts_tablet_manager.cc:1397] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:27:59.267243 10066 log.cc:824] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:59.267565 10068 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Bootstrap starting.
I20260926 07:27:59.270814 10068 tablet_bootstrap.cc:654] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:59.271886 10068 log.cc:824] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:59.273736 10066 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: No bootstrap required, opened a new log
I20260926 07:27:59.273993 10066 ts_tablet_manager.cc:1397] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260926 07:27:59.274094 10068 tablet_bootstrap.cc:492] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: No bootstrap required, opened a new log
I20260926 07:27:59.274317 10068 ts_tablet_manager.cc:1397] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:27:59.278813 10065 raft_consensus.cc:348] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.279260 10065 raft_consensus.cc:374] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:59.279363 10065 raft_consensus.cc:729] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6576e30eaac644bfbc4a2a612b70b887, State: Initialized, Role: FOLLOWER
I20260926 07:27:59.279954 10065 consensus_queue.cc:260] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.282267 10065 ts_tablet_manager.cc:1428] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 07:27:59.282212 10068 raft_consensus.cc:348] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.282543 10035 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
I20260926 07:27:59.282598 10068 raft_consensus.cc:374] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:59.282688 10068 raft_consensus.cc:729] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425211964e7a4614a119f70d6f77fbc3, State: Initialized, Role: FOLLOWER
I20260926 07:27:59.283389 10068 consensus_queue.cc:260] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.285949 9886 heartbeater.cc:503] Master 127.2.147.254:46117 was elected leader, sending a full tablet report...
I20260926 07:27:59.286696 10068 ts_tablet_manager.cc:1428] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260926 07:27:59.286522 10066 raft_consensus.cc:348] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.286909 10066 raft_consensus.cc:374] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:59.287949 10066 raft_consensus.cc:729] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98a232158b547f38305ad1fb7546748, State: Initialized, Role: FOLLOWER
I20260926 07:27:59.288609 10066 consensus_queue.cc:260] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.290349 10066 ts_tablet_manager.cc:1428] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260926 07:27:59.299103 9888 tablet.cc:2369] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:59.390122 10037 tablet.cc:2369] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:59.480854 9746 tablet.cc:2369] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:59.516794 10072 raft_consensus.cc:482] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:59.517084 10072 raft_consensus.cc:504] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.518756 10072 leader_election.cc:288] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619), b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:59.526300 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.519199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:60409 (local address 127.2.147.195:36619)
0926 07:27:59.519446 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.519455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.520459 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 07:27:59.520722 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.520728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.520818 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.520891 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:59.521986 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.522858 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.524647 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.525397 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.525581 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.525650 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.525943 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.526067 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 07:27:59.526333 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.519288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:35729 (local address 127.2.147.193:46847)
0926 07:27:59.519551 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.519560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.520353 (+ 793us) server_negotiation.cc:407] Connection header received
0926 07:27:59.520615 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.520622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.520707 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.520803 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:59.521923 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.522794 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.524647 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.525414 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.525586 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.525665 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.525975 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.526103 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 07:27:59.526501 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.519152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36619 (local address 127.2.147.194:60409)
0926 07:27:59.520354 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.520375 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.520517 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.521037 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.521040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.521058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:59.521812 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.521822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.522983 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.522989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.524508 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.524519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.525602 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.525608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.525764 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.525820 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.525919 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.526050 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":19,"thread_start_us":790,"threads_started":1}
I20260926 07:27:59.526772 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.519087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:46847 (local address 127.2.147.194:35729)
0926 07:27:59.520254 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:59.520267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:59.520420 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:59.520854 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:59.520858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:59.520881 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:59.521771 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.521787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.522929 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.522934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.524501 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:59.524512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.525653 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.525659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:59.525831 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.525853 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:27:59.525950 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:27:59.526083 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":19,"thread_start_us":876,"threads_started":1}
I20260926 07:27:59.527544 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b5390527ce437c98b3735afa7832bc" candidate_uuid: "425211964e7a4614a119f70d6f77fbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576e30eaac644bfbc4a2a612b70b887" is_pre_election: true
I20260926 07:27:59.527689 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b5390527ce437c98b3735afa7832bc" candidate_uuid: "425211964e7a4614a119f70d6f77fbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98a232158b547f38305ad1fb7546748" is_pre_election: true
I20260926 07:27:59.528132 9698 raft_consensus.cc:2454] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 425211964e7a4614a119f70d6f77fbc3 in term 0.
I20260926 07:27:59.528132 9989 raft_consensus.cc:2454] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 425211964e7a4614a119f70d6f77fbc3 in term 0.
I20260926 07:27:59.528832 9776 leader_election.cc:302] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425211964e7a4614a119f70d6f77fbc3, 6576e30eaac644bfbc4a2a612b70b887; no voters:
I20260926 07:27:59.529211 10072 raft_consensus.cc:2781] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:59.529361 10072 raft_consensus.cc:482] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:59.529470 10072 raft_consensus.cc:3037] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:59.531702 10072 raft_consensus.cc:504] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.532624 10072 leader_election.cc:288] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [CANDIDATE]: Term 1 election: Requested vote from peers 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619), b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847)
I20260926 07:27:59.533066 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b5390527ce437c98b3735afa7832bc" candidate_uuid: "425211964e7a4614a119f70d6f77fbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576e30eaac644bfbc4a2a612b70b887"
I20260926 07:27:59.533092 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b5390527ce437c98b3735afa7832bc" candidate_uuid: "425211964e7a4614a119f70d6f77fbc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98a232158b547f38305ad1fb7546748"
I20260926 07:27:59.533247 9989 raft_consensus.cc:3037] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:59.533265 9698 raft_consensus.cc:3037] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:59.535704 9698 raft_consensus.cc:2454] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 425211964e7a4614a119f70d6f77fbc3 in term 1.
I20260926 07:27:59.535908 9989 raft_consensus.cc:2454] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 425211964e7a4614a119f70d6f77fbc3 in term 1.
I20260926 07:27:59.536078 9775 leader_election.cc:302] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425211964e7a4614a119f70d6f77fbc3, b98a232158b547f38305ad1fb7546748; no voters:
I20260926 07:27:59.536346 10072 raft_consensus.cc:2781] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:59.536722 10072 raft_consensus.cc:686] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 1 LEADER]: Becoming Leader. State: Replica: 425211964e7a4614a119f70d6f77fbc3, State: Running, Role: LEADER
I20260926 07:27:59.537225 10072 consensus_queue.cc:237] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } }
I20260926 07:27:59.547778 8778 catalog_manager.cc:5495] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "425211964e7a4614a119f70d6f77fbc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:59.604305 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.599325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.2.147.194:33103)
0926 07:27:59.599521 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.599528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.599547 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:27:59.599664 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.599670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.599762 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.599903 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:59.600244 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.600983 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.602241 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.602623 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.602745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.603163 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:59.603696 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:59.603887 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.603999 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.604057 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:59.612599 9698 raft_consensus.cc:1261] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Refusing update from remote peer 425211964e7a4614a119f70d6f77fbc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:59.612599 9989 raft_consensus.cc:1261] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Refusing update from remote peer 425211964e7a4614a119f70d6f77fbc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:59.613512 10076 consensus_queue.cc:1035] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:59.613840 10072 consensus_queue.cc:1035] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:59.625280 10079 mvcc.cc:205] Tried to move back new op lower bound from 7333509855684841472 to 7333509855388381184. Current Snapshot: MvccSnapshot[applied={T|T < 7333509855684841472}]
I20260926 07:27:59.640674 8778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:59.640954 8778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:59.649858 8778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5beffe5f32114bd492e0e472af2042d5: Sending DeleteTablet for 3 replicas of tablet 70b5390527ce437c98b3735afa7832bc
I20260926 07:27:59.651299 9969 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b5390527ce437c98b3735afa7832bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:59 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37240
I20260926 07:27:59.651535 9820 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b5390527ce437c98b3735afa7832bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:59 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:41160
I20260926 07:27:59.651502 9678 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b5390527ce437c98b3735afa7832bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:59 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:56758
I20260926 07:27:59.651870 9819 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b5390527ce437c98b3735afa7832bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:59 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:41160
I20260926 07:27:59.652374 10091 tablet_replica.cc:330] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: stopping tablet replica
I20260926 07:27:59.652961 10091 raft_consensus.cc:2229] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:59.653594 10091 raft_consensus.cc:2258] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:59.654019 10090 tablet_replica.cc:330] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: stopping tablet replica
I20260926 07:27:59.654289 10092 tablet_replica.cc:330] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: stopping tablet replica
I20260926 07:27:59.654475 10090 raft_consensus.cc:2229] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:59.654641 10092 raft_consensus.cc:2229] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:59.654934 10090 raft_consensus.cc:2258] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:59.657076 10091 ts_tablet_manager.cc:1905] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:59.657228 10090 ts_tablet_manager.cc:1905] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:59.664292 10091 ts_tablet_manager.cc:1918] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:59.664449 10091 log.cc:1196] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/70b5390527ce437c98b3735afa7832bc
I20260926 07:27:59.665009 10091 ts_tablet_manager.cc:1939] T 70b5390527ce437c98b3735afa7832bc P b98a232158b547f38305ad1fb7546748: Deleting consensus metadata
I20260926 07:27:59.655186 10092 raft_consensus.cc:2258] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:59.666016 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 8746
I20260926 07:27:59.668130 10092 ts_tablet_manager.cc:1905] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:59.668308 10090 ts_tablet_manager.cc:1918] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:59.668529 10090 log.cc:1196] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/70b5390527ce437c98b3735afa7832bc
I20260926 07:27:59.669078 10090 ts_tablet_manager.cc:1939] T 70b5390527ce437c98b3735afa7832bc P 6576e30eaac644bfbc4a2a612b70b887: Deleting consensus metadata
I20260926 07:27:59.670672 8765 catalog_manager.cc:4841] TS 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619): tablet 70b5390527ce437c98b3735afa7832bc (table test-table [id=6842f70717904b9496b0711808f13ed0]) successfully deleted
I20260926 07:27:59.671295 8808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:27:59.675669 10092 ts_tablet_manager.cc:1918] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:59.675824 10092 log.cc:1196] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/70b5390527ce437c98b3735afa7832bc
I20260926 07:27:59.676343 10092 ts_tablet_manager.cc:1939] T 70b5390527ce437c98b3735afa7832bc P 425211964e7a4614a119f70d6f77fbc3: Deleting consensus metadata
I20260926 07:27:59.683185 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.653585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.2.147.253:45863)
0926 07:27:59.653735 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.653741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.665282 (+ 11541us) server_negotiation.cc:407] Connection header received
0926 07:27:59.665352 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.665357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.665436 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.665558 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.667845 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.668990 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.670809 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.671497 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.671624 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.676199 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.676208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.676216 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.676281 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.680641 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.681284 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.681290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.681294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.681417 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.681773 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.681778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.681781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.682037 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.682307 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.682643 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.682838 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:27:59.695981 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.665175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.2.147.252:35391)
0926 07:27:59.665356 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.665362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.683335 (+ 17973us) server_negotiation.cc:407] Connection header received
0926 07:27:59.683584 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.683591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.683667 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.683746 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.685760 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.686443 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.687751 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.688148 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.688292 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.690630 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.690634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.690637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.690684 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.693565 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.694145 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.694152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.694155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.694282 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.694644 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.694649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.694652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.694900 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.695172 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.695437 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.695625 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:27:59.832551 8764 catalog_manager.cc:4841] TS b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847): tablet 70b5390527ce437c98b3735afa7832bc (table test-table [id=6842f70717904b9496b0711808f13ed0]) successfully deleted
I20260926 07:27:59.833048 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 8746
W20260926 07:27:59.854295 9773 connection.cc:535] server connection from 127.0.0.1:41160 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:27:59.877617 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.859868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.2.147.252:35391)
0926 07:27:59.860079 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.860085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.860105 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:27:59.860605 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.860611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.860683 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.860783 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.862589 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.863388 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.865290 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.865980 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.866084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.870452 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.870458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.870463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.870501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.874227 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.875155 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.875161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.875165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.875340 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.875794 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.875799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.875802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.876159 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.876373 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.877073 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.877278 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:27:59.877625 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.858534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.2.147.253:45863)
0926 07:27:59.858814 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.858823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.860038 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 07:27:59.860194 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.860200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.860272 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.860378 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.862338 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.863523 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.865345 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.866040 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.866117 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.870452 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.870461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.870465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.870501 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.874369 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.875155 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.875161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.875165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.875376 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.875791 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.875795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.875798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.876177 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.876386 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.877052 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.877278 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:28:00.636714 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.636099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:59607)
0926 07:28:00.636456 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.636556 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":181,"threads_started":1}
W20260926 07:28:00.637084 9744 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:46117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:00.652761 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.652129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:42735)
0926 07:28:00.652492 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.652589 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":200,"threads_started":1}
W20260926 07:28:00.653133 9886 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:46117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:00.653585 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.652871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:53383)
0926 07:28:00.653329 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.653443 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":259,"threads_started":1}
W20260926 07:28:00.653884 10035 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:46117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.148676 10104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beffe5f32114bd492e0e472af2042d5)
I20260926 07:28:01.148943 10104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } }
I20260926 07:28:01.150235 10104 leader_election.cc:288] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117), 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253:45863)
I20260926 07:28:01.151553 10108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beffe5f32114bd492e0e472af2042d5)
I20260926 07:28:01.151670 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.150297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48786)
0926 07:28:01.151038 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.151322 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":339,"threads_started":1}
I20260926 07:28:01.151899 10108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } }
W20260926 07:28:01.152031 9550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.153635 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.153231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48792)
0926 07:28:01.153397 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.153498 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:28:01.153885 9550 leader_election.cc:334] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117): Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.154625 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.153538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48804)
0926 07:28:01.154065 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.154233 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":444,"threads_started":1}
W20260926 07:28:01.154928 9361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.155154 10108 leader_election.cc:288] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117), 6427cb5879ab4e1b92cf3afea868a21b (127.2.147.252:35391)
I20260926 07:28:01.156885 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.156498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48818)
0926 07:28:01.156657 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.156748 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:28:01.157065 9361 leader_election.cc:334] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117): Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.159773 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.150448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45863 (local address 127.0.0.1:47482)
0926 07:28:01.152010 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.152030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.152185 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.152821 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.152824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.152845 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.153183 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.153196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.155919 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.155925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.157771 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.157782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.159020 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.159026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.159173 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.159195 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.159328 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.159450 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"thread_start_us":1300,"threads_started":1}
I20260926 07:28:01.159828 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.150582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.2.147.253:45863)
0926 07:28:01.151268 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.151275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.152142 (+ 867us) server_negotiation.cc:407] Connection header received
0926 07:28:01.152374 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.152381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.152475 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.152601 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.153408 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.155671 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.157918 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.158858 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.159002 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.159088 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.159357 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.159480 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":525,"thread_start_us":367,"threads_started":1}
I20260926 07:28:01.160456 9387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6427cb5879ab4e1b92cf3afea868a21b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" is_pre_election: true
I20260926 07:28:01.160730 9387 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6427cb5879ab4e1b92cf3afea868a21b in term 1.
I20260926 07:28:01.161162 9552 leader_election.cc:302] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38c78ec5d8ab42c4b782a03ef8f3a497, 6427cb5879ab4e1b92cf3afea868a21b; no voters: 5beffe5f32114bd492e0e472af2042d5
I20260926 07:28:01.161568 10104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:28:01.161659 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.154560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:35391 (local address 127.0.0.1:57796)
0926 07:28:01.155307 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.155328 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.155460 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.156093 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.156098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.156127 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.156738 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.156748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.157971 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.157977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.159532 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.159542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.160918 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.160923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.161047 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.161068 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.161162 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.161287 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":26,"thread_start_us":336,"threads_started":1}
I20260926 07:28:01.161731 10104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5beffe5f32114bd492e0e472af2042d5)
I20260926 07:28:01.161863 10104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:28:01.163287 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.154722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.2.147.252:35391)
0926 07:28:01.155742 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.155751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.155784 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:28:01.155838 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.155843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.155931 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.156075 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.156888 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.157851 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.159675 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.160591 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.160726 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.160790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.161186 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.161308 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":897,"thread_start_us":766,"threads_started":1}
I20260926 07:28:01.163810 9576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6427cb5879ab4e1b92cf3afea868a21b" is_pre_election: true
I20260926 07:28:01.164065 10104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } }
I20260926 07:28:01.164453 9576 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38c78ec5d8ab42c4b782a03ef8f3a497 in current term 2: Already voted for candidate 6427cb5879ab4e1b92cf3afea868a21b in this term.
I20260926 07:28:01.164717 10104 leader_election.cc:288] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 election: Requested vote from peers 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117), 38c78ec5d8ab42c4b782a03ef8f3a497 (127.2.147.253:45863)
I20260926 07:28:01.165148 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.164818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48824)
0926 07:28:01.164994 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.165062 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 07:28:01.165135 9387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6427cb5879ab4e1b92cf3afea868a21b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497"
I20260926 07:28:01.165344 9387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:28:01.165282 9363 leader_election.cc:302] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38c78ec5d8ab42c4b782a03ef8f3a497; no voters: 5beffe5f32114bd492e0e472af2042d5, 6427cb5879ab4e1b92cf3afea868a21b
I20260926 07:28:01.166076 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.165769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48834)
0926 07:28:01.165902 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.165980 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:28:01.166232 9550 leader_election.cc:334] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117): Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
I20260926 07:28:01.167397 9387 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6427cb5879ab4e1b92cf3afea868a21b in term 2.
I20260926 07:28:01.167586 10108 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:01.167737 9552 leader_election.cc:302] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38c78ec5d8ab42c4b782a03ef8f3a497, 6427cb5879ab4e1b92cf3afea868a21b; no voters: 5beffe5f32114bd492e0e472af2042d5
I20260926 07:28:01.168035 10104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:28:01.168545 10104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [term 2 LEADER]: Becoming Leader. State: Replica: 6427cb5879ab4e1b92cf3afea868a21b, State: Running, Role: LEADER
I20260926 07:28:01.168902 10104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } }
I20260926 07:28:01.170550 10114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6427cb5879ab4e1b92cf3afea868a21b. Latest consensus state: current_term: 2 leader_uuid: "6427cb5879ab4e1b92cf3afea868a21b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:28:01.170835 10114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:01.171458 10116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:01.174711 10116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:01.175563 10116 catalog_manager.cc:1259] Loaded cluster ID: 52ba795d1f8b47639a872d597d6e00c3
I20260926 07:28:01.175657 10116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:01.176749 10116 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:01.177539 10116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Loaded TSK: 0
I20260926 07:28:01.178314 10116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:28:01.199936 9566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:01.206555 9566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:01.210378 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.210028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48856)
0926 07:28:01.210173 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.210253 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:28:01.210445 9387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [term 2 FOLLOWER]: Refusing update from remote peer 6427cb5879ab4e1b92cf3afea868a21b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 07:28:01.211265 10114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:28:01.211421 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.210994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48870)
0926 07:28:01.211160 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.211277 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:28:01.211607 9550 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b -> Peer 5beffe5f32114bd492e0e472af2042d5 (127.2.147.254:46117): Couldn't send request to peer 5beffe5f32114bd492e0e472af2042d5. Status: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:28:01.213770 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6427cb5879ab4e1b92cf3afea868a21b. Latest consensus state: current_term: 2 leader_uuid: "6427cb5879ab4e1b92cf3afea868a21b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:28:01.214058 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:28:01.215489 10114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6427cb5879ab4e1b92cf3afea868a21b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:28:01.215754 10114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:01.225864 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6427cb5879ab4e1b92cf3afea868a21b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5beffe5f32114bd492e0e472af2042d5" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46117 } } peers { permanent_uuid: "38c78ec5d8ab42c4b782a03ef8f3a497" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45863 } attrs { promote: false } } peers { permanent_uuid: "6427cb5879ab4e1b92cf3afea868a21b" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 35391 } attrs { promote: false } } }
I20260926 07:28:01.226109 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c78ec5d8ab42c4b782a03ef8f3a497 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:28:01.237390 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.231234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.2.147.193:46847)
0926 07:28:01.231667 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.231672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.231687 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:28:01.231759 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.231764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.231836 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.231938 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.232387 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.233390 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.235643 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.236364 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.236511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.236618 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.236945 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.237074 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":236,"threads_started":1}
I20260926 07:28:01.237423 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.230835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:46847 (local address 127.0.0.1:56774)
0926 07:28:01.231383 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.231396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.231482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.231966 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.231971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.231993 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.232249 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.232257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.233561 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.233568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.235478 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.235491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.236578 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.236583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.236785 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.236808 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.236921 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.237046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":21,"thread_start_us":282,"threads_started":1}
I20260926 07:28:01.238330 9678 tablet_service.cc:1467] Processing CreateTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 (DEFAULT_TABLE table=test-table [id=980052481d4e45ba8ac0d6443a380947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:01.238720 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610eaf738d634cae9d42e1aa4e9ef383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:01.239676 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.229984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33103 (local address 127.0.0.1:41184)
0926 07:28:01.230597 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.230616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.230728 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.231651 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.231657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.231685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.231905 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.231918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.234219 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.234231 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.236936 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.236955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.238984 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.238989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.239176 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.239236 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.239331 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.239444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":38}
I20260926 07:28:01.239930 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.230146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.2.147.194:33103)
0926 07:28:01.230561 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.230581 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.230688 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:28:01.230895 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.230903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.231006 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.231638 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.232096 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.234014 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.237176 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.238827 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.238980 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.239109 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.239361 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.239544 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":264,"threads_started":1}
I20260926 07:28:01.240814 9820 tablet_service.cc:1467] Processing CreateTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 (DEFAULT_TABLE table=test-table [id=980052481d4e45ba8ac0d6443a380947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:01.241215 9820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610eaf738d634cae9d42e1aa4e9ef383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:01.245545 10127 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Bootstrap starting.
I20260926 07:28:01.247413 10127 tablet_bootstrap.cc:654] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:01.247803 10128 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Bootstrap starting.
I20260926 07:28:01.249444 10128 tablet_bootstrap.cc:654] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:01.249490 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.230396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36619 (local address 127.0.0.1:37248)
0926 07:28:01.231252 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.231273 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.231374 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.243702 (+ 12328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.243706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.243731 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.244009 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.244017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.245066 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.245071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.247059 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.247072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.248721 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.248727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.248924 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.248959 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.249062 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.249202 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722}
I20260926 07:28:01.249583 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.230945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.2.147.195:36619)
0926 07:28:01.243425 (+ 12480us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.243429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.243444 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:28:01.243498 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.243501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.243550 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.243678 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.244142 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.244911 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.247290 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.248521 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.248666 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.248764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.249104 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.249246 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12386,"thread_start_us":12867,"threads_started":1}
I20260926 07:28:01.249593 10127 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: No bootstrap required, opened a new log
I20260926 07:28:01.249723 10127 ts_tablet_manager.cc:1397] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 07:28:01.250375 9969 tablet_service.cc:1467] Processing CreateTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 (DEFAULT_TABLE table=test-table [id=980052481d4e45ba8ac0d6443a380947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:01.250782 9969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610eaf738d634cae9d42e1aa4e9ef383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:01.250823 10127 raft_consensus.cc:348] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.251036 10127 raft_consensus.cc:374] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:01.251133 10127 raft_consensus.cc:729] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98a232158b547f38305ad1fb7546748, State: Initialized, Role: FOLLOWER
I20260926 07:28:01.251410 10127 consensus_queue.cc:260] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.252192 10127 ts_tablet_manager.cc:1428] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:01.259620 10133 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Bootstrap starting.
I20260926 07:28:01.259959 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.256299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:46509)
0926 07:28:01.259526 (+ 3227us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.259707 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3084,"thread_start_us":2916,"threads_started":1}
I20260926 07:28:01.260691 10128 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: No bootstrap required, opened a new log
I20260926 07:28:01.260846 10128 ts_tablet_manager.cc:1397] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 07:28:01.261287 10133 tablet_bootstrap.cc:654] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:01.261499 10128 raft_consensus.cc:348] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.261665 10128 raft_consensus.cc:374] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:01.261787 10128 raft_consensus.cc:729] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425211964e7a4614a119f70d6f77fbc3, State: Initialized, Role: FOLLOWER
I20260926 07:28:01.261989 10128 consensus_queue.cc:260] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.262632 10128 ts_tablet_manager.cc:1428] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 07:28:01.263804 10133 tablet_bootstrap.cc:492] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: No bootstrap required, opened a new log
I20260926 07:28:01.264000 10133 ts_tablet_manager.cc:1397] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:28:01.264978 10133 raft_consensus.cc:348] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.265270 10133 raft_consensus.cc:374] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:01.265368 10133 raft_consensus.cc:729] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6576e30eaac644bfbc4a2a612b70b887, State: Initialized, Role: FOLLOWER
I20260926 07:28:01.265661 10133 consensus_queue.cc:260] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.267988 9885 heartbeater.cc:503] Master 127.2.147.252:35391 was elected leader, sending a full tablet report...
I20260926 07:28:01.270766 10133 ts_tablet_manager.cc:1428] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 07:28:01.271055 10034 heartbeater.cc:503] Master 127.2.147.252:35391 was elected leader, sending a full tablet report...
I20260926 07:28:01.272087 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.269140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:55641)
0926 07:28:01.271714 (+ 2574us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.271861 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2392,"thread_start_us":2264,"threads_started":1}
I20260926 07:28:01.273265 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.263179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:39159)
0926 07:28:01.272899 (+ 9720us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.273053 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9554,"thread_start_us":9431,"threads_started":1}
I20260926 07:28:01.273720 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.272694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:37349)
0926 07:28:01.273186 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.273491 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260926 07:28:01.274967 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.274654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:59589)
0926 07:28:01.274809 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.274875 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 07:28:01.340723 10131 raft_consensus.cc:482] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:01.340977 10131 raft_consensus.cc:504] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.342490 10131 leader_election.cc:288] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103), 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:28:01.348841 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.342742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:46735 (local address 127.2.147.194:33103)
0926 07:28:01.342974 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.342982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.343005 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:28:01.343250 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.343258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.343342 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.343438 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.344250 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.345068 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.347004 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.347933 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.348106 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.348183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.348454 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.348571 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 07:28:01.348867 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.342679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33103 (local address 127.2.147.193:46735)
0926 07:28:01.342922 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.342943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.343081 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.343487 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.343492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.343526 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.343787 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.343796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.345190 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.345195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.346864 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.346875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.348105 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.348111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.348292 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.348314 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.348423 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.348554 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":2}
I20260926 07:28:01.349086 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.342958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:57761 (local address 127.2.147.195:36619)
0926 07:28:01.343181 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.343190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.343749 (+ 559us) server_negotiation.cc:407] Connection header received
0926 07:28:01.343940 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.343944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.343999 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.344089 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:01.344670 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.345585 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.347320 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.348135 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.348309 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.348382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.348700 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.348825 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:28:01.349368 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.342893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36619 (local address 127.2.147.193:57761)
0926 07:28:01.343651 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.343664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:01.343802 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:01.344103 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:01.344107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:01.344127 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:01.344492 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.344505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.345721 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.345724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.347163 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:01.347178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.348345 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.348351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:01.348515 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.348537 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:28:01.348676 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:28:01.348800 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":17,"thread_start_us":506,"threads_started":1}
I20260926 07:28:01.349687 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610eaf738d634cae9d42e1aa4e9ef383" candidate_uuid: "b98a232158b547f38305ad1fb7546748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425211964e7a4614a119f70d6f77fbc3" is_pre_election: true
I20260926 07:28:01.349982 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610eaf738d634cae9d42e1aa4e9ef383" candidate_uuid: "b98a232158b547f38305ad1fb7546748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576e30eaac644bfbc4a2a612b70b887" is_pre_election: true
I20260926 07:28:01.350075 9840 raft_consensus.cc:2454] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98a232158b547f38305ad1fb7546748 in term 0.
I20260926 07:28:01.350231 9989 raft_consensus.cc:2454] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98a232158b547f38305ad1fb7546748 in term 0.
I20260926 07:28:01.350801 9634 leader_election.cc:302] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 425211964e7a4614a119f70d6f77fbc3, 6576e30eaac644bfbc4a2a612b70b887, b98a232158b547f38305ad1fb7546748; no voters:
I20260926 07:28:01.351248 10131 raft_consensus.cc:2781] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:01.351377 10131 raft_consensus.cc:482] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:01.351471 10131 raft_consensus.cc:3037] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:01.353423 10131 raft_consensus.cc:504] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.353973 10131 leader_election.cc:288] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [CANDIDATE]: Term 1 election: Requested vote from peers 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103), 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619)
I20260926 07:28:01.354305 9840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610eaf738d634cae9d42e1aa4e9ef383" candidate_uuid: "b98a232158b547f38305ad1fb7546748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425211964e7a4614a119f70d6f77fbc3"
I20260926 07:28:01.354480 9840 raft_consensus.cc:3037] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:01.354497 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610eaf738d634cae9d42e1aa4e9ef383" candidate_uuid: "b98a232158b547f38305ad1fb7546748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6576e30eaac644bfbc4a2a612b70b887"
I20260926 07:28:01.354696 9989 raft_consensus.cc:3037] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:01.356647 9989 raft_consensus.cc:2454] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98a232158b547f38305ad1fb7546748 in term 1.
I20260926 07:28:01.356681 9840 raft_consensus.cc:2454] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98a232158b547f38305ad1fb7546748 in term 1.
I20260926 07:28:01.357033 9633 leader_election.cc:302] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425211964e7a4614a119f70d6f77fbc3, b98a232158b547f38305ad1fb7546748; no voters:
I20260926 07:28:01.357343 10131 raft_consensus.cc:2781] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:01.357757 10131 raft_consensus.cc:686] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 1 LEADER]: Becoming Leader. State: Replica: b98a232158b547f38305ad1fb7546748, State: Running, Role: LEADER
I20260926 07:28:01.357998 9743 heartbeater.cc:503] Master 127.2.147.252:35391 was elected leader, sending a full tablet report...
I20260926 07:28:01.358152 10131 consensus_queue.cc:237] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } }
I20260926 07:28:01.358593 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.358237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:53859)
0926 07:28:01.358377 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.358472 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:28:01.361382 9565 catalog_manager.cc:5495] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98a232158b547f38305ad1fb7546748 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "b98a232158b547f38305ad1fb7546748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98a232158b547f38305ad1fb7546748" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46847 } health_report { overall_health: HEALTHY } } }
I20260926 07:28:01.391909 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.386730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.2.147.193:46847)
0926 07:28:01.386935 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.386941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.386960 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:28:01.387164 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.387168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.387290 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.387475 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:01.387796 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.388477 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.389852 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.390266 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.390392 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.390768 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:01.391374 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:01.391537 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.391585 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.391659 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:28:01.395557 9989 raft_consensus.cc:1261] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Refusing update from remote peer b98a232158b547f38305ad1fb7546748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:01.395844 9840 raft_consensus.cc:1261] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Refusing update from remote peer b98a232158b547f38305ad1fb7546748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:01.396270 10141 consensus_queue.cc:1035] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6576e30eaac644bfbc4a2a612b70b887" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:01.396623 10131 consensus_queue.cc:1035] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "425211964e7a4614a119f70d6f77fbc3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:01.399572 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.398890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:53861)
0926 07:28:01.399164 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.399325 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 07:28:01.407330 10144 mvcc.cc:205] Tried to move back new op lower bound from 7333509862990041088 to 7333509862847057920. Current Snapshot: MvccSnapshot[applied={T|T < 7333509862990041088}]
I20260926 07:28:01.414714 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.402721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:45985)
0926 07:28:01.414390 (+ 11669us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.414518 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11555}
I20260926 07:28:01.416503 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.415912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.194:50021)
0926 07:28:01.416225 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.416323 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260926 07:28:01.420715 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.419838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:40551)
0926 07:28:01.420344 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.420476 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
I20260926 07:28:01.428207 9565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:01.428591 9565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:01.431551 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.431045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.195:40419)
0926 07:28:01.431296 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.431392 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 07:28:01.437821 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.435005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.2.147.193:35455)
0926 07:28:01.437515 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.437636 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2341}
I20260926 07:28:01.445642 9565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6427cb5879ab4e1b92cf3afea868a21b: Sending DeleteTablet for 3 replicas of tablet 610eaf738d634cae9d42e1aa4e9ef383
I20260926 07:28:01.446652 9969 tablet_service.cc:1514] Processing DeleteTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:01 UTC) from {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:37248
I20260926 07:28:01.447916 9678 tablet_service.cc:1514] Processing DeleteTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:01 UTC) from {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:56774
I20260926 07:28:01.448199 9677 tablet_service.cc:1514] Processing DeleteTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:01 UTC) from {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:56774
I20260926 07:28:01.448799 9820 tablet_service.cc:1514] Processing DeleteTablet for tablet 610eaf738d634cae9d42e1aa4e9ef383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:01 UTC) from {username='oryx', principal='oryx/127.2.147.252@KRBTEST.COM'} at 127.0.0.1:41184
I20260926 07:28:01.449633 10155 tablet_replica.cc:330] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: stopping tablet replica
I20260926 07:28:01.449857 10155 raft_consensus.cc:2229] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:01.450127 10155 raft_consensus.cc:2258] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:01.450765 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9540
I20260926 07:28:01.451167 10155 ts_tablet_manager.cc:1905] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:01.457442 10155 ts_tablet_manager.cc:1918] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:01.457598 10155 log.cc:1196] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/610eaf738d634cae9d42e1aa4e9ef383
I20260926 07:28:01.458041 10155 ts_tablet_manager.cc:1939] T 610eaf738d634cae9d42e1aa4e9ef383 P 6576e30eaac644bfbc4a2a612b70b887: Deleting consensus metadata
I20260926 07:28:01.459357 10157 tablet_replica.cc:330] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: stopping tablet replica
I20260926 07:28:01.459208 9553 catalog_manager.cc:4841] TS 6576e30eaac644bfbc4a2a612b70b887 (127.2.147.195:36619): tablet 610eaf738d634cae9d42e1aa4e9ef383 (table test-table [id=980052481d4e45ba8ac0d6443a380947]) successfully deleted
I20260926 07:28:01.459542 10157 raft_consensus.cc:2229] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:01.459841 10157 raft_consensus.cc:2258] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:01.460368 10156 tablet_replica.cc:330] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: stopping tablet replica
I20260926 07:28:01.460601 10156 raft_consensus.cc:2229] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:01.461184 10156 raft_consensus.cc:2258] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:01.461284 10157 ts_tablet_manager.cc:1905] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:01.462553 10156 ts_tablet_manager.cc:1905] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:01.468308 10157 ts_tablet_manager.cc:1918] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:01.468488 10157 log.cc:1196] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/610eaf738d634cae9d42e1aa4e9ef383
I20260926 07:28:01.469048 10157 ts_tablet_manager.cc:1939] T 610eaf738d634cae9d42e1aa4e9ef383 P 425211964e7a4614a119f70d6f77fbc3: Deleting consensus metadata
I20260926 07:28:01.469496 10156 ts_tablet_manager.cc:1918] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:01.469681 10156 log.cc:1196] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/610eaf738d634cae9d42e1aa4e9ef383
I20260926 07:28:01.470196 10156 ts_tablet_manager.cc:1939] T 610eaf738d634cae9d42e1aa4e9ef383 P b98a232158b547f38305ad1fb7546748: Deleting consensus metadata
I20260926 07:28:01.470333 9552 catalog_manager.cc:4841] TS 425211964e7a4614a119f70d6f77fbc3 (127.2.147.194:33103): tablet 610eaf738d634cae9d42e1aa4e9ef383 (table test-table [id=980052481d4e45ba8ac0d6443a380947]) successfully deleted
I20260926 07:28:01.471345 9552 catalog_manager.cc:4841] TS b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847): tablet 610eaf738d634cae9d42e1aa4e9ef383 (table test-table [id=980052481d4e45ba8ac0d6443a380947]) successfully deleted
W20260926 07:28:01.471884 9552 catalog_manager.cc:4808] TS b98a232158b547f38305ad1fb7546748 (127.2.147.193:46847): delete failed for tablet 610eaf738d634cae9d42e1aa4e9ef383 because the tablet was not found. No further retry: Not found: Tablet not found: 610eaf738d634cae9d42e1aa4e9ef383
I20260926 07:28:01.477377 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.452533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.2.147.193:46847)
0926 07:28:01.452713 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.452719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.461966 (+ 9247us) server_negotiation.cc:407] Connection header received
0926 07:28:01.462154 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.462160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.462237 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.462352 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:01.464012 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.464751 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.466550 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.467022 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.467164 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.470488 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:01.470502 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:01.470523 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:01.470566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:01.474565 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.475233 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.475244 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.475247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.475438 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.475801 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.475806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.475809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.476248 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:01.476526 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.476791 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.477005 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:28:01.478056 9738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:01.579109 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9540
I20260926 07:28:01.597440 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9757
I20260926 07:28:01.610810 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.598148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.2.147.194:33103)
0926 07:28:01.598336 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.598344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.598397 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:28:01.598588 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.598594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.598666 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.598785 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:01.600058 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.600797 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.602100 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.602529 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.602668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.605234 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:01.605255 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:01.605270 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:01.605300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:01.608224 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.608859 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.608869 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.608873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.609019 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.609371 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.609377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.609381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.609713 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:01.609996 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.610276 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.610518 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:28:01.611469 9880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:01.665481 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.665050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48880)
0926 07:28:01.665223 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.665327 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:28:01.666419 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.666120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48892)
0926 07:28:01.666237 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:01.666300 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:28:01.732059 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9757
I20260926 07:28:01.753182 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9906
I20260926 07:28:01.766037 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:01.753835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.2.147.195:36619)
0926 07:28:01.754006 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:01.754012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:01.754056 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:28:01.754288 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:01.754293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:01.754373 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:01.754473 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:01.755854 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.756575 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.757764 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:01.758155 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:01.758281 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:01.760806 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:01.760821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:01.760839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:01.760868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:01.763497 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.764090 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.764103 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.764105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.764217 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:01.764571 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:01.764577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:01.764580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:01.764913 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:01.765261 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:01.765547 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:28:01.765744 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:01.766601 10029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:01.858019 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9906
I20260926 07:28:01.875711 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9239
I20260926 07:28:01.876741 9407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:01.990748 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9239
I20260926 07:28:02.011132 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 9338
I20260926 07:28:02.012223 9596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:02.132989 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:02.132503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46117 (local address 127.0.0.1:48896)
0926 07:28:02.132688 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:02.132770 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:28:02.132969 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 9338
2026-09-26T07:28:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9258 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:02 dist-test-slave-1dwd krb5kdc[10166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:02 dist-test-slave-1dwd krb5kdc[10166](info): set up 2 sockets
Sep 26 07:28:02 dist-test-slave-1dwd krb5kdc[10166](info): commencing operation
krb5kdc: starting...
W20260926 07:28:03.208178 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:28:03 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab.
Entry for principal oryx/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab.
I20260926 07:28:03.400476 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45453
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab
--principal=oryx/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:03.721405 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:03.721828 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:03.722010 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:03.734377 10179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:03.734480 10179 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:28:03.734542 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:03.734622 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:03.734692 10179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:03.734769 10179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:03.744275 10179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:45453
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.254.keytab
--principal=oryx/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:03.745464 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:03.746745 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:03.757978 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:03.757941 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:03.758009 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:03.758682 10179 server_base.cc:1034] running on GCE node
I20260926 07:28:03.759483 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:03.760793 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:03.762013 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790407683761960 us; error 25 us; skew 500 ppm
I20260926 07:28:03.762343 10179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:03 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407683, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:03.765412 10179 init.cc:377] Logged in from keytab as oryx/127.2.147.254@KRBTEST.COM (short username oryx)
I20260926 07:28:03.766855 10179 webserver.cc:460] Webserver started at http://127.2.147.254:45883/ using document root <none> and password file <none>
I20260926 07:28:03.767573 10179 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:03.767700 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:03.768034 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:03.770917 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "062e1698d18245cdb7b5f23af770c515"
format_stamp: "Formatted at 2026-09-26 07:28:03 on dist-test-slave-1dwd"
server_key: "4ad2b7a991e7a0336f83b4d0513ae976"
server_key_iv: "f80a82bce8b83149ba7b553fae2eae25"
server_key_version: "encryptionkey@0"
I20260926 07:28:03.771764 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "062e1698d18245cdb7b5f23af770c515"
format_stamp: "Formatted at 2026-09-26 07:28:03 on dist-test-slave-1dwd"
server_key: "4ad2b7a991e7a0336f83b4d0513ae976"
server_key_iv: "f80a82bce8b83149ba7b553fae2eae25"
server_key_version: "encryptionkey@0"
I20260926 07:28:03.776816 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:28:03.780040 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:03.781284 10179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:03.781458 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "062e1698d18245cdb7b5f23af770c515"
format_stamp: "Formatted at 2026-09-26 07:28:03 on dist-test-slave-1dwd"
server_key: "4ad2b7a991e7a0336f83b4d0513ae976"
server_key_iv: "f80a82bce8b83149ba7b553fae2eae25"
server_key_version: "encryptionkey@0"
I20260926 07:28:03.781672 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:03.796556 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:03.800185 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:03.800457 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:03.820629 10179 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:45453
I20260926 07:28:03.820639 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:45453 every 8 connection(s)
I20260926 07:28:03.821919 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:03.825366 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:03.827055 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10179
I20260926 07:28:03.827280 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:28:03.828114 2639 external_mini_cluster.cc:1428] Setting key 60f89d83bbcd8a1945a99efa7b10c35c
I20260926 07:28:03.839134 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Bootstrap starting.
Sep 26 07:28:03 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:03.842805 10256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:03.843844 10256 log.cc:824] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:03.846439 10256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: No bootstrap required, opened a new log
I20260926 07:28:03.850119 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:03.830018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.2.147.254:45453)
0926 07:28:03.830851 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:03.830862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:03.830912 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:28:03.832496 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:03.832518 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:03.832914 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:03.833210 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:03.834907 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:03.836218 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:03.837918 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:03.838528 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:03.838645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:03.843321 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:03.843329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:03.843343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:03.843375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:03.846293 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:03.846978 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:03.846985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:03.846991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:03.847145 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:03.847536 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:03.847547 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:03.847551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:03.848075 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:03.848395 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:03.848780 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 07:28:03.849180 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":408,"threads_started":1}
I20260926 07:28:03.854094 10256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } }
I20260926 07:28:03.854427 10256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:03.854521 10256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062e1698d18245cdb7b5f23af770c515, State: Initialized, Role: FOLLOWER
I20260926 07:28:03.855070 10256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } }
I20260926 07:28:03.855381 10256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:03.855551 10256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:03.855718 10256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:03.857781 10256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } }
I20260926 07:28:03.858240 10256 leader_election.cc:302] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 062e1698d18245cdb7b5f23af770c515; no voters:
I20260926 07:28:03.858786 10256 leader_election.cc:288] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:03.858901 10261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:03.859642 10261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [term 1 LEADER]: Becoming Leader. State: Replica: 062e1698d18245cdb7b5f23af770c515, State: Running, Role: LEADER
I20260926 07:28:03.860143 10261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } }
I20260926 07:28:03.860807 10256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:03.862839 10262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "062e1698d18245cdb7b5f23af770c515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } } }
I20260926 07:28:03.862846 10263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 062e1698d18245cdb7b5f23af770c515. Latest consensus state: current_term: 1 leader_uuid: "062e1698d18245cdb7b5f23af770c515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062e1698d18245cdb7b5f23af770c515" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45453 } } }
I20260926 07:28:03.863332 10262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:03.863425 10263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515 [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:03.864995 10266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:03.869858 10266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:03.878350 10266 catalog_manager.cc:1347] Generated new cluster ID: 282759d82974461cbc255f781d3c1904
I20260926 07:28:03.878455 10266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:03.892235 10266 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:03.893441 10266 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:03.899574 10266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Generated new TSK 0
I20260926 07:28:03.900529 10266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
I20260926 07:28:03.961030 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45453
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:04.285136 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:04.285444 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:04.285647 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:04.292944 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:04.293077 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:04.293223 10282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:28:04.293349 10282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:04.302603 10282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:04.303890 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:04.305160 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:04.317759 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:04.317787 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:04.317855 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:04.318415 10282 server_base.cc:1034] running on GCE node
I20260926 07:28:04.319068 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:04.319940 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:04.321215 10282 hybrid_clock.cc:648] HybridClock initialized: now 1790407684321127 us; error 59 us; skew 500 ppm
I20260926 07:28:04.321501 10282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:04 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407684, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:04.324673 10282 init.cc:377] Logged in from keytab as oryx/127.2.147.193@KRBTEST.COM (short username oryx)
I20260926 07:28:04.326056 10282 webserver.cc:460] Webserver started at http://127.2.147.193:38367/ using document root <none> and password file <none>
I20260926 07:28:04.326675 10282 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:04.326757 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:04.327014 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:04.329789 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "83ad7c0fe42eea5c557e929a23bcad8e"
server_key_iv: "ce6b3449fea898a2284750c3cf621886"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.330524 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "83ad7c0fe42eea5c557e929a23bcad8e"
server_key_iv: "ce6b3449fea898a2284750c3cf621886"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.335322 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:28:04.338587 10297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:04.339884 10282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:28:04.340006 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "83ad7c0fe42eea5c557e929a23bcad8e"
server_key_iv: "ce6b3449fea898a2284750c3cf621886"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.340173 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:04.354977 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:04.359007 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:04.359256 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:04.361910 10282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:04.362075 10282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:04.362219 10282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:04.362313 10282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:28:04 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407684, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:04.384889 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.362216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:48607 (local address 127.2.147.254:45453)
0926 07:28:04.362800 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:04.362807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:04.364018 (+ 1211us) server_negotiation.cc:407] Connection header received
0926 07:28:04.366332 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:04.366338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:04.366436 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:04.366556 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:04.369084 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.370337 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.372830 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.373232 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.373367 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.377413 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:04.377417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:04.377421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:04.377451 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:04.381226 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.381913 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.381934 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.381939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.382111 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.382538 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.382545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.382551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.382952 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:04.383167 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:04.384049 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 07:28:04.384352 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
I20260926 07:28:04.385622 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.362913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.193:48607)
0926 07:28:04.363846 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:04.363902 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:04.366057 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:04.367165 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:04.367179 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:04.368078 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:04.368879 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.368898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.370509 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.370514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.372669 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.372694 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.373452 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.373462 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.373689 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.374464 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:04.374484 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:04.377142 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:04.381370 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.381376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.381386 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.381777 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.382228 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.382231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.382233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.382421 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.383119 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:04.383129 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:04.383567 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 07:28:04.383933 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 07:28:04.384326 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":434,"threads_started":1}
I20260926 07:28:04.406805 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.391994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:46845 (local address 127.2.147.254:45453)
0926 07:28:04.392208 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:04.392216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:04.392801 (+ 585us) server_negotiation.cc:407] Connection header received
0926 07:28:04.392988 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:04.392995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:04.393081 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:04.393188 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:04.394702 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.395741 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.397737 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.398137 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.398276 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.400495 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:04.400504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:04.400511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:04.400546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:04.404113 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.405015 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.405022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.405026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.405153 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.405602 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.405605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.405607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.405839 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:04.405936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:04.406455 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 07:28:04.406599 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:28:04.407469 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.391930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.193:46845)
0926 07:28:04.392664 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:04.392688 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:04.392842 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:04.393232 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:04.393237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:04.393941 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:04.394504 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.394518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.395891 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.395896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.397568 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.397579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.398313 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.398319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.398473 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.399084 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:04.399100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:04.400201 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:04.404237 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.404247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.404252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.404890 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.405264 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.405271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.405277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.405484 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.405947 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:04.405954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:04.406181 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 07:28:04.406392 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 07:28:04.406789 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":359,"threads_started":1}
I20260926 07:28:04.414953 10282 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45763
I20260926 07:28:04.414971 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45763 every 8 connection(s)
I20260926 07:28:04.416164 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:04.418143 10418 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45453
I20260926 07:28:04.418359 10418 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:04.419063 10418 heartbeater.cc:511] Master 127.2.147.254:45453 requested a full tablet report, sending...
I20260926 07:28:04.420841 10211 ts_manager.cc:194] Registered new tserver with Master: dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763)
I20260926 07:28:04.422149 10211 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:48607
I20260926 07:28:04.423964 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10282
I20260926 07:28:04.424146 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:04.424907 2639 external_mini_cluster.cc:1428] Setting key a9875625ce04c0767f54b8b0099687a4
WARNING: no policy specified for oryx/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
I20260926 07:28:04.467862 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45453
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:04.776803 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:04.777092 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:04.777259 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:04.784126 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:04.784271 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:04.784405 10422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:28:04.784541 10422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:04.793785 10422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:04.794939 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:04.796211 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:04.808347 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:04.808327 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:04.808774 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:04.808997 10422 server_base.cc:1034] running on GCE node
I20260926 07:28:04.809675 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:04.817562 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:04.818800 10422 hybrid_clock.cc:648] HybridClock initialized: now 1790407684818751 us; error 76 us; skew 500 ppm
I20260926 07:28:04.819108 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:04 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407684, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:04.822467 10422 init.cc:377] Logged in from keytab as oryx/127.2.147.194@KRBTEST.COM (short username oryx)
I20260926 07:28:04.824002 10422 webserver.cc:460] Webserver started at http://127.2.147.194:34713/ using document root <none> and password file <none>
I20260926 07:28:04.824816 10422 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:04.824919 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:04.825378 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:04.829144 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "60dc28a1550b4218be2df6b4a91246e9"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "a368fde85d6f4a5cb1628cbe6da83b87"
server_key_iv: "d66717bf11a283003abd7ac6e9e3e73f"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.830088 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "60dc28a1550b4218be2df6b4a91246e9"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "a368fde85d6f4a5cb1628cbe6da83b87"
server_key_iv: "d66717bf11a283003abd7ac6e9e3e73f"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.835821 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 07:28:04.839715 10437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:04.841046 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:04.841204 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "60dc28a1550b4218be2df6b4a91246e9"
format_stamp: "Formatted at 2026-09-26 07:28:04 on dist-test-slave-1dwd"
server_key: "a368fde85d6f4a5cb1628cbe6da83b87"
server_key_iv: "d66717bf11a283003abd7ac6e9e3e73f"
server_key_version: "encryptionkey@0"
I20260926 07:28:04.841408 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:04.874275 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:04.878218 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:04.878499 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:04.881109 10422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:04.881223 10422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:04.881384 10422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:04.881462 10422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:28:04 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407684, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:04.905175 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.881981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55863 (local address 127.2.147.254:45453)
0926 07:28:04.882174 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:04.882179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:04.883445 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 07:28:04.885520 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:04.885526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:04.885619 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:04.885767 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:04.888346 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.889441 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.892832 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.893433 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.893567 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.898860 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:04.898866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:04.898878 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:04.898921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:04.901672 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.902711 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.902714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.902716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.902824 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.903380 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.903386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.903392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.903744 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:04.903893 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:04.904624 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 07:28:04.904842 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:28:04.906119 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.882431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.194:55863)
0926 07:28:04.883270 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:04.883322 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:04.885242 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:04.886322 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:04.886330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:04.887159 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:04.888164 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.888182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.889612 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.889620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.892653 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.892674 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.893729 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.893739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.894079 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.894851 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:04.894889 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:04.898568 (+ 3679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:04.901804 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.901816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.901837 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.902572 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.902925 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.902932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.902938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.903229 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.903874 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:04.903881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:04.904209 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 07:28:04.904522 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 07:28:04.905002 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":358,"threads_started":1}
I20260926 07:28:04.926352 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.911628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:38267 (local address 127.2.147.254:45453)
0926 07:28:04.911800 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:04.911808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:04.913392 (+ 1584us) server_negotiation.cc:407] Connection header received
0926 07:28:04.913578 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:04.913584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:04.913655 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:04.913752 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:04.914583 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.915356 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.917265 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.918007 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.918151 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.920118 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:04.920123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:04.920126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:04.920158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:04.923305 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.924311 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.924319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.924322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.924444 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:04.924928 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:04.924933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:04.924936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:04.925232 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:04.925343 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:04.925813 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 07:28:04.926014 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:28:04.926527 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:04.911791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.194:38267)
0926 07:28:04.913295 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:04.913313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:04.913456 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:04.913772 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:04.913776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:04.914202 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:04.914455 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.914463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.915484 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.915495 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.917118 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:04.917129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:04.918218 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:04.918225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:04.918395 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:04.919004 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:04.919020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:04.919911 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:04.923449 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.923456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.923459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.924175 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.924562 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:04.924568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:04.924573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:04.924783 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:04.925420 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:04.925424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:04.925567 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:28:04.925729 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 07:28:04.926040 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":1228,"threads_started":1}
I20260926 07:28:04.937215 10422 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:34703
I20260926 07:28:04.937239 10556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:34703 every 8 connection(s)
I20260926 07:28:04.938418 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:04.940325 10557 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45453
I20260926 07:28:04.940505 10557 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:04.941052 10557 heartbeater.cc:511] Master 127.2.147.254:45453 requested a full tablet report, sending...
I20260926 07:28:04.942196 10211 ts_manager.cc:194] Registered new tserver with Master: 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703)
I20260926 07:28:04.942971 10211 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:55863
I20260926 07:28:04.948144 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10422
I20260926 07:28:04.948277 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:04.948648 2639 external_mini_cluster.cc:1428] Setting key 8942d7c2774560769b48a694478211ad
WARNING: no policy specified for oryx/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
I20260926 07:28:04.994297 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45453
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:05.305302 10561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:05.305559 10561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:05.305670 10561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:05.312922 10561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:05.313023 10561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:05.313112 10561 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:28:05.313187 10561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:05.322227 10561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.147.254:45453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:05.323341 10561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:05.324518 10561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:05.336892 10567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:05.337118 10566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:05.337082 10569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:05.337920 10561 server_base.cc:1034] running on GCE node
I20260926 07:28:05.338475 10561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:05.339330 10561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:05.340533 10561 hybrid_clock.cc:648] HybridClock initialized: now 1790407685340478 us; error 32 us; skew 500 ppm
I20260926 07:28:05.340765 10561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:05 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407685, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:05.343735 10561 init.cc:377] Logged in from keytab as oryx/127.2.147.195@KRBTEST.COM (short username oryx)
I20260926 07:28:05.345162 10561 webserver.cc:460] Webserver started at http://127.2.147.195:44951/ using document root <none> and password file <none>
I20260926 07:28:05.345849 10561 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:05.345952 10561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:05.346252 10561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:05.348973 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "72530ca0003341fe8720fa5e2cc9295f"
format_stamp: "Formatted at 2026-09-26 07:28:05 on dist-test-slave-1dwd"
server_key: "f7f8d8afb49f1ee7e8c467aa67ac9b2e"
server_key_iv: "e878f9d5a4de3e7c3a87fe0c94f62f43"
server_key_version: "encryptionkey@0"
I20260926 07:28:05.349711 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "72530ca0003341fe8720fa5e2cc9295f"
format_stamp: "Formatted at 2026-09-26 07:28:05 on dist-test-slave-1dwd"
server_key: "f7f8d8afb49f1ee7e8c467aa67ac9b2e"
server_key_iv: "e878f9d5a4de3e7c3a87fe0c94f62f43"
server_key_version: "encryptionkey@0"
I20260926 07:28:05.354425 10561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:28:05.357601 10576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:05.358764 10561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:05.358925 10561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "72530ca0003341fe8720fa5e2cc9295f"
format_stamp: "Formatted at 2026-09-26 07:28:05 on dist-test-slave-1dwd"
server_key: "f7f8d8afb49f1ee7e8c467aa67ac9b2e"
server_key_iv: "e878f9d5a4de3e7c3a87fe0c94f62f43"
server_key_version: "encryptionkey@0"
I20260926 07:28:05.359135 10561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:05.370422 10561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:05.373966 10561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:05.374228 10561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:05.376704 10561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:05.376807 10561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:05.376991 10561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:05.377068 10561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:28:05 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407685, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:05.400200 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:05.377141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:34253 (local address 127.2.147.254:45453)
0926 07:28:05.377323 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:05.377329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:05.378735 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 07:28:05.381005 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:05.381011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:05.381091 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:05.381207 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:05.383208 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.384315 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.386959 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.387718 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.387883 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:05.392446 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:05.392453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:05.392459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:05.392500 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:05.395880 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:05.396807 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:05.396816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:05.396822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:05.396975 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:05.397531 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:05.397539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:05.397542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:05.397958 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:05.398101 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:05.399467 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0926 07:28:05.399751 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:28:05.401540 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:05.377719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.195:34253)
0926 07:28:05.378549 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:05.378598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:05.380665 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:05.381428 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:05.381436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:05.382133 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:05.382979 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:05.383009 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.384463 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.384469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:05.386737 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:05.386749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.387961 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.387968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:05.388148 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:05.388881 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:05.388919 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:05.392161 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:05.396014 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:05.396023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:05.396041 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:05.396654 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:05.397108 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:05.397115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:05.397121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:05.397386 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:05.398087 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:05.398094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:05.398569 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 07:28:05.398895 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 07:28:05.399655 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":359,"threads_started":1}
I20260926 07:28:05.423131 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:05.408507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.2.147.195:41593)
0926 07:28:05.409373 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:05.409397 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:05.409587 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:05.409885 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:05.409889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:05.410350 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:05.410618 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:05.410626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.412011 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.412017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:05.413011 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:05.413021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.414078 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.414082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:05.414254 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:05.414976 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:05.414996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:05.415976 (+ 980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:05.419733 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:05.419742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:05.419749 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:05.420390 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:05.420812 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:05.420818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:05.420822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:05.421033 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:05.421723 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:05.421734 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:05.421939 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 07:28:05.422108 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 07:28:05.422432 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":514,"threads_started":1}
I20260926 07:28:05.423476 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:05.408420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:41593 (local address 127.2.147.254:45453)
0926 07:28:05.408587 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:05.408594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:05.409478 (+ 884us) server_negotiation.cc:407] Connection header received
0926 07:28:05.409693 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:05.409697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:05.409766 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:05.409862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:05.410749 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.411885 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.413147 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.413709 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.413850 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:05.416169 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:05.416175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:05.416178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:05.416210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:05.419549 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:05.420534 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:05.420539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:05.420542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:05.420679 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:05.421165 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:05.421170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:05.421173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:05.421550 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:05.422852 (+ 1302us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:05.422960 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:28:05.423099 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:05.425093 10418 heartbeater.cc:503] Master 127.2.147.254:45453 was elected leader, sending a full tablet report...
I20260926 07:28:05.429266 10561 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:33221
I20260926 07:28:05.429327 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:33221 every 8 connection(s)
I20260926 07:28:05.430359 10561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:05.432082 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10561
I20260926 07:28:05.432265 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:05.432390 10696 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45453
I20260926 07:28:05.432585 10696 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:05.432809 2639 external_mini_cluster.cc:1428] Setting key ddd2f2859eb534cdc2ee4d804d86b104
I20260926 07:28:05.433115 10696 heartbeater.cc:511] Master 127.2.147.254:45453 requested a full tablet report, sending...
I20260926 07:28:05.434504 10211 ts_manager.cc:194] Registered new tserver with Master: 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:05.435436 10211 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:34253
I20260926 07:28:05.435727 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:28:05 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:28:05 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:05 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:05.945436 10557 heartbeater.cc:503] Master 127.2.147.254:45453 was elected leader, sending a full tablet report...
I20260926 07:28:06.437695 10696 heartbeater.cc:503] Master 127.2.147.254:45453 was elected leader, sending a full tablet report...
W20260926 07:28:06.531430 10704 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.052s user 0.007s sys 0.007s
W20260926 07:28:06.532626 10305 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:05.473550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.2.147.254:45453)
0926 07:28:05.473772 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:05.473780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:05.474047 (+ 267us) server_negotiation.cc:407] Connection header received
0926 07:28:05.474262 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:05.474266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:05.474324 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:05.474427 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:05.475833 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.476822 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.478013 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:05.478435 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:05.478570 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.532295 (+1053725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60758: BlockingRecv error: recv got EOF from 127.0.0.1:60758 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 07:28:06 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:06.549476 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.536783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.2.147.254:45453)
0926 07:28:06.537001 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.537006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.537254 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:28:06.537385 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.537388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.537458 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.537539 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:06.538650 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.539416 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.540518 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.540894 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.540981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.544717 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:06.544721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:06.544724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:06.544752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:06.547374 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:06.547941 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:06.547944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:06.547947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:06.548050 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:06.548348 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:06.548351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:06.548353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:06.548588 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:06.548789 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.548976 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.549134 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:28:06.553149 10211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:06.559957 10211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:06.596972 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.572017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.2.147.193:45763)
0926 07:28:06.589985 (+ 17968us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.589997 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.590032 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:28:06.590152 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.590164 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.590485 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.590724 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:06.591275 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.592748 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.595013 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.595986 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.596157 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.596369 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.596644 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.596774 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17781,"thread_start_us":17511,"threads_started":1}
I20260926 07:28:06.597103 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.571612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.0.0.1:53914)
0926 07:28:06.583029 (+ 11417us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:06.583073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:06.583304 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:06.590718 (+ 7414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:06.590726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:06.590753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:06.591063 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.591071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.592867 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.592875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.594861 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.594876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.596218 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.596228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.596410 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.596464 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 07:28:06.596598 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:28:06.596769 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11119,"thread_start_us":10952,"threads_started":1}
I20260926 07:28:06.599836 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.572062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.2.147.194:34703)
0926 07:28:06.591367 (+ 19305us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.591373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.591393 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:28:06.591459 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.591463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.591693 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.591914 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:06.592516 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.593965 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.597415 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.598823 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.598961 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.599171 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.599383 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.599554 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19143,"thread_start_us":19010,"threads_started":1}
I20260926 07:28:06.599850 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.571795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:34703 (local address 127.0.0.1:44208)
0926 07:28:06.589235 (+ 17440us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:06.589257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:06.589403 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:06.591922 (+ 2519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:06.591927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:06.591953 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:06.592334 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.592344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.594130 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.594136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.596333 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.596347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.599020 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.599024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.599134 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.599155 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:28:06.599327 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 07:28:06.599446 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17261,"mutex_wait_us":21,"thread_start_us":17145,"threads_started":1}
I20260926 07:28:06.600531 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.574440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.2.147.195:33221)
0926 07:28:06.589575 (+ 15135us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.589588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.590244 (+ 656us) server_negotiation.cc:407] Connection header received
0926 07:28:06.590464 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.590473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.591368 (+ 895us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.591558 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:06.592175 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.593651 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.596461 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.598184 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.599292 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.599483 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.600088 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.600249 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14943,"thread_start_us":14798,"threads_started":1}
I20260926 07:28:06.601356 10348 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5113b1665042f787455379237221dc (DEFAULT_TABLE table=test-table [id=0c4a702cf277449d8dc965428357f5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:06.602828 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5113b1665042f787455379237221dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:06.603176 10486 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5113b1665042f787455379237221dc (DEFAULT_TABLE table=test-table [id=0c4a702cf277449d8dc965428357f5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:06.604461 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5113b1665042f787455379237221dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:06.607666 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.573935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:33221 (local address 127.0.0.1:46990)
0926 07:28:06.590037 (+ 16102us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:06.590052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:06.590167 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:06.591563 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:06.591568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:06.591591 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:06.591990 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.592003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.593856 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.593861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.596333 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.596347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.599633 (+ 3286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.599639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.599792 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.599821 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:28:06.599907 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:28:06.607367 (+ 7460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15992,"mutex_wait_us":50,"thread_start_us":15855,"threads_started":1}
I20260926 07:28:06.611408 10626 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5113b1665042f787455379237221dc (DEFAULT_TABLE table=test-table [id=0c4a702cf277449d8dc965428357f5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:06.612704 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5113b1665042f787455379237221dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:06.618299 10722 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Bootstrap starting.
I20260926 07:28:06.619558 10721 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Bootstrap starting.
I20260926 07:28:06.621963 10722 tablet_bootstrap.cc:654] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:06.622869 10721 tablet_bootstrap.cc:654] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:06.623234 10722 log.cc:824] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:06.624148 10721 log.cc:824] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:06.627054 10725 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Bootstrap starting.
I20260926 07:28:06.627302 10721 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: No bootstrap required, opened a new log
I20260926 07:28:06.627432 10722 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: No bootstrap required, opened a new log
I20260926 07:28:06.627552 10721 ts_tablet_manager.cc:1397] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:28:06.627729 10722 ts_tablet_manager.cc:1397] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 07:28:06.630451 10725 tablet_bootstrap.cc:654] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:06.631677 10725 log.cc:824] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:06.634752 10725 tablet_bootstrap.cc:492] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: No bootstrap required, opened a new log
I20260926 07:28:06.635037 10725 ts_tablet_manager.cc:1397] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260926 07:28:06.635434 10722 raft_consensus.cc:348] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.635787 10722 raft_consensus.cc:374] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:06.635905 10722 raft_consensus.cc:729] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Initialized, Role: FOLLOWER
I20260926 07:28:06.636463 10722 consensus_queue.cc:260] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.637521 10721 raft_consensus.cc:348] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.637949 10721 raft_consensus.cc:374] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:06.638000 10722 ts_tablet_manager.cc:1428] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260926 07:28:06.638051 10721 raft_consensus.cc:729] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Initialized, Role: FOLLOWER
I20260926 07:28:06.638600 10721 consensus_queue.cc:260] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.640185 10721 ts_tablet_manager.cc:1428] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 07:28:06.645970 10725 raft_consensus.cc:348] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.646333 10725 raft_consensus.cc:374] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:06.646441 10725 raft_consensus.cc:729] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Initialized, Role: FOLLOWER
I20260926 07:28:06.646948 10725 consensus_queue.cc:260] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.648473 10725 ts_tablet_manager.cc:1428] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260926 07:28:06.669068 10419 tablet.cc:2369] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:28:06.682757 10697 tablet.cc:2369] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:28:06.691013 10558 tablet.cc:2369] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:06.883978 10729 raft_consensus.cc:482] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:06.884332 10729 raft_consensus.cc:504] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.886139 10729 leader_election.cc:288] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703)
I20260926 07:28:06.895284 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.886580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:50895 (local address 127.2.147.193:45763)
0926 07:28:06.886788 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.886795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.887600 (+ 805us) server_negotiation.cc:407] Connection header received
0926 07:28:06.887812 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.887817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.887901 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.888000 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:06.888824 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.889930 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.892921 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.894154 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.894314 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.894383 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.894906 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.895026 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:28:06.895284 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.886663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:52649 (local address 127.2.147.194:34703)
0926 07:28:06.886836 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.886841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.887593 (+ 752us) server_negotiation.cc:407] Connection header received
0926 07:28:06.887807 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.887812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.887903 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.888000 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:06.888829 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.889928 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.892921 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.894154 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.894315 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.894383 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.894906 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.895026 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:06.895522 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.886434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.2.147.195:50895)
0926 07:28:06.887496 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:06.887533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:06.887672 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:06.888029 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:06.888034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:06.888085 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:06.888684 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.888694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.890061 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.890064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.892759 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.892773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.894389 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.894401 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.894687 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.894775 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 07:28:06.894869 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:28:06.895015 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":309,"spinlock_wait_cycles":33664,"thread_start_us":607,"threads_started":1}
I20260926 07:28:06.895562 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.886405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:34703 (local address 127.2.147.195:52649)
0926 07:28:06.887467 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:06.887490 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:06.887646 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:06.888029 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:06.888034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:06.888079 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:06.888683 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.888694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.890059 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.890064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.892769 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:06.892776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.894416 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.894421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:06.894689 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.894782 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:28:06.894869 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:28:06.895015 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":373,"spinlock_wait_cycles":768,"thread_start_us":658,"threads_started":1}
I20260926 07:28:06.896570 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5113b1665042f787455379237221dc" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9" is_pre_election: true
I20260926 07:28:06.897125 10512 raft_consensus.cc:2454] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 0.
I20260926 07:28:06.897374 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5113b1665042f787455379237221dc" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" is_pre_election: true
I20260926 07:28:06.897795 10580 leader_election.cc:302] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, 72530ca0003341fe8720fa5e2cc9295f; no voters:
I20260926 07:28:06.897853 10373 raft_consensus.cc:2454] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 0.
I20260926 07:28:06.898177 10729 raft_consensus.cc:2781] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:06.898355 10729 raft_consensus.cc:482] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:06.898486 10729 raft_consensus.cc:3037] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:06.900668 10729 raft_consensus.cc:504] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.901175 10729 leader_election.cc:288] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 election: Requested vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703)
I20260926 07:28:06.901608 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5113b1665042f787455379237221dc" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
I20260926 07:28:06.901657 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5113b1665042f787455379237221dc" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9"
I20260926 07:28:06.901790 10373 raft_consensus.cc:3037] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:06.901836 10512 raft_consensus.cc:3037] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:06.903930 10373 raft_consensus.cc:2454] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 1.
I20260926 07:28:06.904052 10512 raft_consensus.cc:2454] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 1.
I20260926 07:28:06.904340 10577 leader_election.cc:302] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72530ca0003341fe8720fa5e2cc9295f, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:06.904596 10729 raft_consensus.cc:2781] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:06.905006 10729 raft_consensus.cc:686] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 1 LEADER]: Becoming Leader. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Running, Role: LEADER
I20260926 07:28:06.905452 10729 consensus_queue.cc:237] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:06.909578 10211 catalog_manager.cc:5495] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f reported cstate change: term changed from 0 to 1, leader changed from <none> to 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195). New cstate: current_term: 1 leader_uuid: "72530ca0003341fe8720fa5e2cc9295f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } health_report { overall_health: HEALTHY } } }
I20260926 07:28:06.959259 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:06.954560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.2.147.195:33221)
0926 07:28:06.954734 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:06.954738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:06.954755 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:28:06.954894 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:06.954898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:06.954983 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:06.955112 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:06.955492 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.956184 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.957408 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:06.957787 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:06.957887 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:06.958239 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:06.958737 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:06.958837 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:06.958964 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 07:28:06.959022 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:28:06.967093 10512 raft_consensus.cc:1261] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Refusing update from remote peer 72530ca0003341fe8720fa5e2cc9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:06.967111 10373 raft_consensus.cc:1261] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Refusing update from remote peer 72530ca0003341fe8720fa5e2cc9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:06.967998 10732 consensus_queue.cc:1035] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:06.968374 10729 consensus_queue.cc:1035] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:06.980381 10737 mvcc.cc:205] Tried to move back new op lower bound from 7333509885808328704 to 7333509885568049152. Current Snapshot: MvccSnapshot[applied={T|T < 7333509885808328704}]
I20260926 07:28:06.984943 10736 mvcc.cc:205] Tried to move back new op lower bound from 7333509885808328704 to 7333509885568049152. Current Snapshot: MvccSnapshot[applied={T|T < 7333509885808328704}]
I20260926 07:28:06.990477 10211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:06.990701 10211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:06.993688 10211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Sending DeleteTablet for 3 replicas of tablet 0a5113b1665042f787455379237221dc
I20260926 07:28:06.994623 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5113b1665042f787455379237221dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:06 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53914
I20260926 07:28:06.994807 10626 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5113b1665042f787455379237221dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:06 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:46990
I20260926 07:28:06.995783 10486 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5113b1665042f787455379237221dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:06 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44208
I20260926 07:28:06.999692 10211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:07.000972 10211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:07.006951 10745 tablet_replica.cc:330] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: stopping tablet replica
I20260926 07:28:07.007426 10745 raft_consensus.cc:2229] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:07.009582 10745 raft_consensus.cc:2258] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.011023 10347 tablet_service.cc:1467] Processing CreateTablet for tablet d403a1e9d67c40639be2edc07d448191 (DEFAULT_TABLE table=test-table [id=51bd565854a74014bedc4e0d56f1b1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.011415 10746 tablet_replica.cc:330] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: stopping tablet replica
I20260926 07:28:07.011799 10746 raft_consensus.cc:2229] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.012044 10487 tablet_service.cc:1467] Processing CreateTablet for tablet d403a1e9d67c40639be2edc07d448191 (DEFAULT_TABLE table=test-table [id=51bd565854a74014bedc4e0d56f1b1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.012288 10746 raft_consensus.cc:2258] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.012622 10487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403a1e9d67c40639be2edc07d448191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.012703 10745 ts_tablet_manager.cc:1905] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.013886 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403a1e9d67c40639be2edc07d448191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.018954 10722 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Bootstrap starting.
I20260926 07:28:07.019778 10745 ts_tablet_manager.cc:1918] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.019982 10745 log.cc:1196] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/0a5113b1665042f787455379237221dc
I20260926 07:28:07.020186 10626 tablet_service.cc:1467] Processing CreateTablet for tablet d403a1e9d67c40639be2edc07d448191 (DEFAULT_TABLE table=test-table [id=51bd565854a74014bedc4e0d56f1b1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.020476 10722 tablet_bootstrap.cc:654] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.020609 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403a1e9d67c40639be2edc07d448191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.020700 10745 ts_tablet_manager.cc:1939] T 0a5113b1665042f787455379237221dc P 72530ca0003341fe8720fa5e2cc9295f: Deleting consensus metadata
I20260926 07:28:07.013077 10744 tablet_replica.cc:330] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: stopping tablet replica
I20260926 07:28:07.022084 10721 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Bootstrap starting.
I20260926 07:28:07.022459 10744 raft_consensus.cc:2229] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.022950 10744 raft_consensus.cc:2258] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.023761 10746 ts_tablet_manager.cc:1905] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.024118 10721 tablet_bootstrap.cc:654] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.024513 10197 catalog_manager.cc:4841] TS 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221): tablet 0a5113b1665042f787455379237221dc (table test-table [id=0c4a702cf277449d8dc965428357f5f5]) successfully deleted
I20260926 07:28:07.026182 10744 ts_tablet_manager.cc:1905] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.030822 10746 ts_tablet_manager.cc:1918] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.031036 10746 log.cc:1196] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/0a5113b1665042f787455379237221dc
I20260926 07:28:07.031271 10725 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Bootstrap starting.
I20260926 07:28:07.031641 10746 ts_tablet_manager.cc:1939] T 0a5113b1665042f787455379237221dc P 60dc28a1550b4218be2df6b4a91246e9: Deleting consensus metadata
I20260926 07:28:07.032625 10744 ts_tablet_manager.cc:1918] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.032761 10744 log.cc:1196] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/0a5113b1665042f787455379237221dc
I20260926 07:28:07.033272 10744 ts_tablet_manager.cc:1939] T 0a5113b1665042f787455379237221dc P dfcdd42b7535473982fa5b3dd3d449d6: Deleting consensus metadata
I20260926 07:28:07.033258 10198 catalog_manager.cc:4841] TS 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703): tablet 0a5113b1665042f787455379237221dc (table test-table [id=0c4a702cf277449d8dc965428357f5f5]) successfully deleted
I20260926 07:28:07.033306 10725 tablet_bootstrap.cc:654] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.034673 10195 catalog_manager.cc:4841] TS dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763): tablet 0a5113b1665042f787455379237221dc (table test-table [id=0c4a702cf277449d8dc965428357f5f5]) successfully deleted
I20260926 07:28:07.043663 10725 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: No bootstrap required, opened a new log
I20260926 07:28:07.043877 10725 ts_tablet_manager.cc:1397] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260926 07:28:07.044648 10721 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: No bootstrap required, opened a new log
I20260926 07:28:07.044795 10722 tablet_bootstrap.cc:492] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: No bootstrap required, opened a new log
I20260926 07:28:07.044885 10721 ts_tablet_manager.cc:1397] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.010s
I20260926 07:28:07.044955 10722 ts_tablet_manager.cc:1397] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.006s
I20260926 07:28:07.044874 10725 raft_consensus.cc:348] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.045342 10725 raft_consensus.cc:374] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.045486 10725 raft_consensus.cc:729] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.045702 10722 raft_consensus.cc:348] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.045702 10721 raft_consensus.cc:348] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.045939 10722 raft_consensus.cc:374] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.046008 10721 raft_consensus.cc:374] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.046063 10722 raft_consensus.cc:729] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.045835 10725 consensus_queue.cc:260] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.046134 10721 raft_consensus.cc:729] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.046377 10722 consensus_queue.cc:260] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.046409 10721 consensus_queue.cc:260] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.046984 10722 ts_tablet_manager.cc:1428] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 07:28:07.047622 10721 ts_tablet_manager.cc:1428] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:28:07.049396 10725 ts_tablet_manager.cc:1428] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:28:07.187860 10732 raft_consensus.cc:482] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:07.188108 10732 raft_consensus.cc:504] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.188825 10732 leader_election.cc:288] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703)
I20260926 07:28:07.189337 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403a1e9d67c40639be2edc07d448191" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9" is_pre_election: true
I20260926 07:28:07.189375 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403a1e9d67c40639be2edc07d448191" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" is_pre_election: true
I20260926 07:28:07.189572 10512 raft_consensus.cc:2454] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 0.
I20260926 07:28:07.189600 10373 raft_consensus.cc:2454] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 0.
I20260926 07:28:07.189996 10577 leader_election.cc:302] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72530ca0003341fe8720fa5e2cc9295f, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.190275 10732 raft_consensus.cc:2781] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:07.190390 10732 raft_consensus.cc:482] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:07.190546 10732 raft_consensus.cc:3037] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.192880 10732 raft_consensus.cc:504] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.193477 10732 leader_election.cc:288] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 election: Requested vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703)
I20260926 07:28:07.193919 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403a1e9d67c40639be2edc07d448191" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
I20260926 07:28:07.193949 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403a1e9d67c40639be2edc07d448191" candidate_uuid: "72530ca0003341fe8720fa5e2cc9295f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9"
I20260926 07:28:07.194128 10512 raft_consensus.cc:3037] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.194137 10373 raft_consensus.cc:3037] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.196954 10373 raft_consensus.cc:2454] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 1.
I20260926 07:28:07.196954 10512 raft_consensus.cc:2454] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72530ca0003341fe8720fa5e2cc9295f in term 1.
I20260926 07:28:07.197382 10580 leader_election.cc:302] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, 72530ca0003341fe8720fa5e2cc9295f, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.197708 10732 raft_consensus.cc:2781] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:07.197919 10732 raft_consensus.cc:686] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 1 LEADER]: Becoming Leader. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Running, Role: LEADER
I20260926 07:28:07.198460 10732 consensus_queue.cc:237] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.201316 10208 catalog_manager.cc:5495] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f reported cstate change: term changed from 0 to 1, leader changed from <none> to 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195). New cstate: current_term: 1 leader_uuid: "72530ca0003341fe8720fa5e2cc9295f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } health_report { overall_health: HEALTHY } } }
Sep 26 07:28:07 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407683, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.254@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:07.259583 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.245895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.2.147.254:45453)
0926 07:28:07.246816 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.246822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.246846 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:28:07.246895 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.246899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.246954 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.247072 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:07.248167 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.248958 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.250147 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.250581 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.250719 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.254259 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:07.254264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:07.254267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:07.254293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:07.256989 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:07.257632 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:07.257639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:07.257642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:07.257750 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:07.258128 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:07.258134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:07.258137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:07.258395 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:07.258543 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.258935 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.259169 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":786,"thread_start_us":591,"threads_started":1}
I20260926 07:28:07.259729 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.245852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45453 (local address 127.0.0.1:60776)
0926 07:28:07.246562 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.246588 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.246751 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.247091 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.247095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.247794 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:07.248015 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.248024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.249087 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.249095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.250011 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.250022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.250740 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.250745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.250910 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.251678 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:07.251695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:07.254073 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:07.257118 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:07.257131 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:07.257140 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:07.257510 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:07.257855 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:07.257860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:07.257863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:07.258022 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:07.258523 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:07.258530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:07.258697 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.258864 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.259332 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":160,"thread_start_us":334,"threads_started":1}
I20260926 07:28:07.265089 10208 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:60776:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:28:07.266160 10208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:07.273105 10347 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea8ccc1857a4e2880a857bb211e758d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d70524313de44823bf02b0eef37b7f4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:28:07.273352 10487 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea8ccc1857a4e2880a857bb211e758d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d70524313de44823bf02b0eef37b7f4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:28:07.273716 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea8ccc1857a4e2880a857bb211e758d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.273859 10487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea8ccc1857a4e2880a857bb211e758d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.274039 10626 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea8ccc1857a4e2880a857bb211e758d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d70524313de44823bf02b0eef37b7f4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:28:07.274608 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea8ccc1857a4e2880a857bb211e758d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.281081 10722 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: Bootstrap starting.
I20260926 07:28:07.281486 10721 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: Bootstrap starting.
I20260926 07:28:07.282817 10722 tablet_bootstrap.cc:654] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.283010 10725 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: Bootstrap starting.
I20260926 07:28:07.283313 10721 tablet_bootstrap.cc:654] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.285265 10725 tablet_bootstrap.cc:654] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.293314 10722 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: No bootstrap required, opened a new log
I20260926 07:28:07.293551 10722 ts_tablet_manager.cc:1397] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260926 07:28:07.294220 10721 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: No bootstrap required, opened a new log
I20260926 07:28:07.294448 10721 ts_tablet_manager.cc:1397] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260926 07:28:07.294450 10722 raft_consensus.cc:348] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.294806 10722 raft_consensus.cc:374] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.294930 10722 raft_consensus.cc:729] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.295305 10722 consensus_queue.cc:260] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.295373 10721 raft_consensus.cc:348] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.295652 10721 raft_consensus.cc:374] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.295751 10721 raft_consensus.cc:729] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.295881 10727 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.295979 10721 consensus_queue.cc:260] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.296213 10727 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.296454 10728 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.296842 10728 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.297338 10721 ts_tablet_manager.cc:1428] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:07.297678 10725 tablet_bootstrap.cc:492] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: No bootstrap required, opened a new log
I20260926 07:28:07.297859 10725 ts_tablet_manager.cc:1397] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260926 07:28:07.298693 10725 raft_consensus.cc:348] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.298980 10725 raft_consensus.cc:374] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.299111 10725 raft_consensus.cc:729] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.299865 10725 consensus_queue.cc:260] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.300406 10732 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.300671 10732 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.301301 10725 ts_tablet_manager.cc:1428] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:07.302889 10722 ts_tablet_manager.cc:1428] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260926 07:28:07.344412 10728 raft_consensus.cc:482] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:07.344682 10728 raft_consensus.cc:504] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.346379 10728 leader_election.cc:288] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:07.354707 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.347436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50105 (local address 127.2.147.195:33221)
0926 07:28:07.347646 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.347653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.348374 (+ 721us) server_negotiation.cc:407] Connection header received
0926 07:28:07.348546 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.348553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.348632 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.348728 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:07.349360 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.350463 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.352901 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.353697 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.353895 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.353969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.354255 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.354387 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:28:07.354776 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.346811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:33221 (local address 127.2.147.193:50105)
0926 07:28:07.348258 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.348276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.348409 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.348766 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.348770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.348802 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:07.349181 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.349193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.350630 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.350635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.352704 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.352717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.353914 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.353920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.354084 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.354109 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.354230 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.354352 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":5,"thread_start_us":830,"threads_started":1}
I20260926 07:28:07.355419 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.346818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60823 (local address 127.2.147.194:34703)
0926 07:28:07.347039 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.347048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.347708 (+ 660us) server_negotiation.cc:407] Connection header received
0926 07:28:07.348047 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.348052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.348127 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.348192 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:07.349052 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.350168 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.352830 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.354138 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.354333 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.354405 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.354958 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.355095 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:28:07.355770 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea8ccc1857a4e2880a857bb211e758d" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f" is_pre_election: true
I20260926 07:28:07.356115 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.346529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:34703 (local address 127.2.147.193:60823)
0926 07:28:07.347581 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.347616 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.347855 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.348372 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.348376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.348436 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:07.348874 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.348884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.350327 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.350332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.352650 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.352664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.354566 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.354572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.354766 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.354800 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.355598 (+ 798us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.355733 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"spinlock_wait_cycles":13696,"thread_start_us":1434,"threads_started":1}
I20260926 07:28:07.356227 10644 raft_consensus.cc:2454] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 0.
I20260926 07:28:07.356747 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea8ccc1857a4e2880a857bb211e758d" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9" is_pre_election: true
I20260926 07:28:07.356993 10512 raft_consensus.cc:2454] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 0.
I20260926 07:28:07.356920 10300 leader_election.cc:302] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72530ca0003341fe8720fa5e2cc9295f, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.357312 10728 raft_consensus.cc:2781] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:07.357456 10728 raft_consensus.cc:482] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:07.357563 10728 raft_consensus.cc:3037] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.359673 10728 raft_consensus.cc:504] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.360251 10728 leader_election.cc:288] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 election: Requested vote from peers 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:07.360608 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea8ccc1857a4e2880a857bb211e758d" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9"
I20260926 07:28:07.360699 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea8ccc1857a4e2880a857bb211e758d" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f"
I20260926 07:28:07.360801 10512 raft_consensus.cc:3037] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.360919 10644 raft_consensus.cc:3037] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.363061 10512 raft_consensus.cc:2454] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 1.
I20260926 07:28:07.363526 10301 leader_election.cc:302] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.363839 10728 raft_consensus.cc:2781] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:07.364316 10644 raft_consensus.cc:2454] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 1.
I20260926 07:28:07.364327 10728 raft_consensus.cc:686] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 LEADER]: Becoming Leader. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Running, Role: LEADER
I20260926 07:28:07.364768 10728 consensus_queue.cc:237] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.367105 10763 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: TxnStatusTablet state changed. Reason: New leader dfcdd42b7535473982fa5b3dd3d449d6. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.367548 10763 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:28:07.368265 10765 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:28:07.368484 10765 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:28:07.368870 10211 catalog_manager.cc:5495] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20260926 07:28:07.398443 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.392753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.0.0.1:53930)
0926 07:28:07.393098 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.393115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.393264 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.393720 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.393724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.393735 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:28:07.393910 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.393918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.394921 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.394929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.396154 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.396165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.396875 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.396880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.397044 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.397093 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.397908 (+ 815us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.397927 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.398008 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.398151 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260926 07:28:07.398468 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.392764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.2.147.193:45763)
0926 07:28:07.392948 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.392954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.393183 (+ 229us) server_negotiation.cc:407] Connection header received
0926 07:28:07.393462 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.393466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.393544 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.393703 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:07.394031 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.394790 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.396283 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.396696 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.396873 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.397281 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.397801 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.397946 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.398024 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.398114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:28:07.402431 10512 raft_consensus.cc:1261] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Refusing update from remote peer dfcdd42b7535473982fa5b3dd3d449d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.402561 10644 raft_consensus.cc:1261] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Refusing update from remote peer dfcdd42b7535473982fa5b3dd3d449d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.403308 10728 consensus_queue.cc:1035] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.403606 10763 consensus_queue.cc:1035] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.417016 10732 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: TxnStatusTablet state changed. Reason: New leader dfcdd42b7535473982fa5b3dd3d449d6. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.417389 10732 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.419699 10727 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: TxnStatusTablet state changed. Reason: New leader dfcdd42b7535473982fa5b3dd3d449d6. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.419999 10727 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.421231 10763 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.421522 10763 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:28:07.422960 10732 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.423223 10732 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 72530ca0003341fe8720fa5e2cc9295f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.430800 10727 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.431027 10727 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P 60dc28a1550b4218be2df6b4a91246e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:28:07.433703 10728 tablet_replica.cc:439] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } } }
I20260926 07:28:07.435290 10728 tablet_replica.cc:442] T 6ea8ccc1857a4e2880a857bb211e758d P dfcdd42b7535473982fa5b3dd3d449d6: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:28:07.492944 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.478824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:34617 (local address 127.2.147.193:45763)
0926 07:28:07.478988 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.478994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.486421 (+ 7427us) server_negotiation.cc:407] Connection header received
0926 07:28:07.486668 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.486676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.486763 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.486853 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:07.487658 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.488880 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.491056 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.491646 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.491719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.492211 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.492319 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.492414 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.492559 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.492631 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:28:07.493139 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.478351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.2.147.195:34617)
0926 07:28:07.486307 (+ 7956us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.486324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.486498 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.487023 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.487028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.487040 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:28:07.487434 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.487445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.489089 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.489094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.490899 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.490911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.491859 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.491863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.491975 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.492025 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.492415 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.492435 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.492527 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.492661 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7777,"thread_start_us":7569,"threads_started":1}
I20260926 07:28:07.502992 10373 raft_consensus.cc:1261] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Refusing update from remote peer 72530ca0003341fe8720fa5e2cc9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.503000 10512 raft_consensus.cc:1261] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Refusing update from remote peer 72530ca0003341fe8720fa5e2cc9295f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.503638 10732 consensus_queue.cc:1035] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.503935 10729 consensus_queue.cc:1035] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.511641 10735 mvcc.cc:205] Tried to move back new op lower bound from 7333509888007958528 to 7333509886767497216. Current Snapshot: MvccSnapshot[applied={T|T < 7333509888007958528}]
I20260926 07:28:07.537984 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.530906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:33221 (local address 127.2.147.193:33333)
0926 07:28:07.531583 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:07.531623 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:07.531759 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:07.532265 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:07.532270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:07.532281 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:28:07.532738 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.532749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.534239 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.534243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.536010 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:07.536022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.536971 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.536975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:07.537088 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.537141 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.537508 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.537521 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:28:07.537573 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:28:07.537662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":434,"threads_started":1}
I20260926 07:28:07.537971 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.530860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:33333 (local address 127.2.147.195:33221)
0926 07:28:07.531019 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.531025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.531706 (+ 681us) server_negotiation.cc:407] Connection header received
0926 07:28:07.532063 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.532069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.532145 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.532233 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:07.533005 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.534131 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.536147 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.536783 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.536898 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.537290 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.537408 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.537495 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.537606 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.537683 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:28:07.573454 10208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:07.573737 10208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:07.576731 10208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Sending DeleteTablet for 3 replicas of tablet d403a1e9d67c40639be2edc07d448191
I20260926 07:28:07.577457 10347 tablet_service.cc:1514] Processing DeleteTablet for tablet d403a1e9d67c40639be2edc07d448191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53914
I20260926 07:28:07.577545 10487 tablet_service.cc:1514] Processing DeleteTablet for tablet d403a1e9d67c40639be2edc07d448191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44208
I20260926 07:28:07.577947 10626 tablet_service.cc:1514] Processing DeleteTablet for tablet d403a1e9d67c40639be2edc07d448191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:46990
I20260926 07:28:07.588543 10781 tablet_replica.cc:330] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: stopping tablet replica
I20260926 07:28:07.588770 10781 raft_consensus.cc:2229] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.589098 10781 raft_consensus.cc:2258] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.590158 10781 ts_tablet_manager.cc:1905] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.592439 10780 tablet_replica.cc:330] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: stopping tablet replica
I20260926 07:28:07.592721 10780 raft_consensus.cc:2229] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.593045 10780 raft_consensus.cc:2258] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.594358 10780 ts_tablet_manager.cc:1905] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.596994 10781 ts_tablet_manager.cc:1918] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:07.597111 10781 log.cc:1196] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/d403a1e9d67c40639be2edc07d448191
I20260926 07:28:07.597532 10781 ts_tablet_manager.cc:1939] T d403a1e9d67c40639be2edc07d448191 P 60dc28a1550b4218be2df6b4a91246e9: Deleting consensus metadata
I20260926 07:28:07.599772 10198 catalog_manager.cc:4841] TS 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703): tablet d403a1e9d67c40639be2edc07d448191 (table test-table [id=51bd565854a74014bedc4e0d56f1b1ec]) successfully deleted
I20260926 07:28:07.602463 10780 ts_tablet_manager.cc:1918] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:07.602633 10780 log.cc:1196] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/d403a1e9d67c40639be2edc07d448191
I20260926 07:28:07.603263 10780 ts_tablet_manager.cc:1939] T d403a1e9d67c40639be2edc07d448191 P dfcdd42b7535473982fa5b3dd3d449d6: Deleting consensus metadata
I20260926 07:28:07.604019 10782 tablet_replica.cc:330] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: stopping tablet replica
I20260926 07:28:07.604228 10782 raft_consensus.cc:2229] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:07.604398 10195 catalog_manager.cc:4841] TS dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763): tablet d403a1e9d67c40639be2edc07d448191 (table test-table [id=51bd565854a74014bedc4e0d56f1b1ec]) successfully deleted
I20260926 07:28:07.604588 10782 raft_consensus.cc:2258] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.605439 10782 ts_tablet_manager.cc:1905] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.610206 10782 ts_tablet_manager.cc:1918] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:07.610335 10782 log.cc:1196] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/d403a1e9d67c40639be2edc07d448191
I20260926 07:28:07.610728 10782 ts_tablet_manager.cc:1939] T d403a1e9d67c40639be2edc07d448191 P 72530ca0003341fe8720fa5e2cc9295f: Deleting consensus metadata
I20260926 07:28:07.611662 10197 catalog_manager.cc:4841] TS 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221): tablet d403a1e9d67c40639be2edc07d448191 (table test-table [id=51bd565854a74014bedc4e0d56f1b1ec]) successfully deleted
Sep 26 07:28:07 dist-test-slave-1dwd krb5kdc[10166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:28:07 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:07 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:28:07.651336 10757 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:07.639440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.2.147.254:45453)
0926 07:28:07.639717 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.639727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.640161 (+ 434us) server_negotiation.cc:407] Connection header received
0926 07:28:07.640337 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.640341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.640403 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.640487 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:07.642129 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.642827 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.644741 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.645195 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.645290 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.651058 (+ 5768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60778: BlockingRecv error: recv got EOF from 127.0.0.1:60778 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 07:28:07 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM
I20260926 07:28:07.669214 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.654075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.2.147.254:45453)
0926 07:28:07.654231 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.654236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.654670 (+ 434us) server_negotiation.cc:407] Connection header received
0926 07:28:07.654817 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.654821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.654876 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.654952 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:07.656478 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.657163 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.658883 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.659349 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.659447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.663914 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:07.663918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:07.663921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:07.663950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:07.666604 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:07.667346 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:07.667351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:07.667355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:07.667486 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:07.667856 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:07.667860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:07.667866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:07.668109 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:07.668359 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.668649 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.668884 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:28:07.672770 10208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:07.673480 10208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:07.680328 10347 tablet_service.cc:1467] Processing CreateTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda (DEFAULT_TABLE table=test-table [id=23773d644816488092ee3fee14b16a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.680533 10626 tablet_service.cc:1467] Processing CreateTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda (DEFAULT_TABLE table=test-table [id=23773d644816488092ee3fee14b16a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.680804 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7625bf8c0bf64dac8e680be12d11ebda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.680932 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7625bf8c0bf64dac8e680be12d11ebda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.683825 10487 tablet_service.cc:1467] Processing CreateTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda (DEFAULT_TABLE table=test-table [id=23773d644816488092ee3fee14b16a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.684249 10487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7625bf8c0bf64dac8e680be12d11ebda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.687537 10725 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Bootstrap starting.
I20260926 07:28:07.687546 10721 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Bootstrap starting.
I20260926 07:28:07.689184 10721 tablet_bootstrap.cc:654] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.689181 10725 tablet_bootstrap.cc:654] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.691085 10722 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Bootstrap starting.
I20260926 07:28:07.692559 10722 tablet_bootstrap.cc:654] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.699683 10721 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: No bootstrap required, opened a new log
I20260926 07:28:07.699842 10721 ts_tablet_manager.cc:1397] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260926 07:28:07.699899 10725 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: No bootstrap required, opened a new log
I20260926 07:28:07.700044 10725 ts_tablet_manager.cc:1397] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260926 07:28:07.700661 10721 raft_consensus.cc:348] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.700737 10725 raft_consensus.cc:348] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.700939 10721 raft_consensus.cc:374] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.700981 10725 raft_consensus.cc:374] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.701061 10721 raft_consensus.cc:729] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.701081 10725 raft_consensus.cc:729] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.701311 10725 consensus_queue.cc:260] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.701328 10721 consensus_queue.cc:260] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.701834 10725 ts_tablet_manager.cc:1428] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 07:28:07.701920 10721 ts_tablet_manager.cc:1428] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:28:07.705317 10722 tablet_bootstrap.cc:492] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: No bootstrap required, opened a new log
I20260926 07:28:07.705469 10722 ts_tablet_manager.cc:1397] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260926 07:28:07.706241 10722 raft_consensus.cc:348] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.706496 10722 raft_consensus.cc:374] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.706601 10722 raft_consensus.cc:729] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.706871 10722 consensus_queue.cc:260] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.707499 10722 ts_tablet_manager.cc:1428] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 07:28:07.787726 10763 raft_consensus.cc:482] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:07.787956 10763 raft_consensus.cc:504] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.788803 10763 leader_election.cc:288] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:07.789341 10646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7625bf8c0bf64dac8e680be12d11ebda" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f" is_pre_election: true
I20260926 07:28:07.789381 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7625bf8c0bf64dac8e680be12d11ebda" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9" is_pre_election: true
I20260926 07:28:07.789601 10646 raft_consensus.cc:2454] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 0.
I20260926 07:28:07.789633 10512 raft_consensus.cc:2454] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 0.
I20260926 07:28:07.790023 10300 leader_election.cc:302] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72530ca0003341fe8720fa5e2cc9295f, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.790325 10763 raft_consensus.cc:2781] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:07.790436 10763 raft_consensus.cc:482] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:07.790570 10763 raft_consensus.cc:3037] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.792757 10763 raft_consensus.cc:504] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.793319 10763 leader_election.cc:288] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 election: Requested vote from peers 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:07.793761 10512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7625bf8c0bf64dac8e680be12d11ebda" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dc28a1550b4218be2df6b4a91246e9"
I20260926 07:28:07.793838 10646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7625bf8c0bf64dac8e680be12d11ebda" candidate_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f"
I20260926 07:28:07.794018 10512 raft_consensus.cc:3037] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.794044 10646 raft_consensus.cc:3037] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:07.796068 10512 raft_consensus.cc:2454] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 1.
I20260926 07:28:07.796486 10301 leader_election.cc:302] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:07.796738 10763 raft_consensus.cc:2781] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:07.797101 10646 raft_consensus.cc:2454] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcdd42b7535473982fa5b3dd3d449d6 in term 1.
I20260926 07:28:07.796918 10763 raft_consensus.cc:686] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 LEADER]: Becoming Leader. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Running, Role: LEADER
I20260926 07:28:07.797467 10763 consensus_queue.cc:237] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.805058 10211 catalog_manager.cc:5495] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20260926 07:28:07.836606 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:07.830263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.2.147.193:45763)
0926 07:28:07.830469 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:07.830476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:07.830631 (+ 155us) server_negotiation.cc:407] Connection header received
0926 07:28:07.830832 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:07.830837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:07.830926 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:07.831022 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:07.831392 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.832516 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.834568 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:07.835305 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:07.835429 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:07.835903 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:07.836046 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:07.836129 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:07.836271 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 07:28:07.836347 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:28:07.839905 10512 raft_consensus.cc:1261] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Refusing update from remote peer dfcdd42b7535473982fa5b3dd3d449d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.840016 10646 raft_consensus.cc:1261] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Refusing update from remote peer dfcdd42b7535473982fa5b3dd3d449d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:07.840525 10763 consensus_queue.cc:1035] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.840773 10728 consensus_queue.cc:1035] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:07.861581 10209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:07.861850 10209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:07.865067 10209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Sending DeleteTablet for 3 replicas of tablet 7625bf8c0bf64dac8e680be12d11ebda
I20260926 07:28:07.865724 10487 tablet_service.cc:1514] Processing DeleteTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44208
I20260926 07:28:07.865762 10347 tablet_service.cc:1514] Processing DeleteTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53914
I20260926 07:28:07.865973 10781 tablet_replica.cc:330] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: stopping tablet replica
I20260926 07:28:07.866024 10626 tablet_service.cc:1514] Processing DeleteTablet for tablet 7625bf8c0bf64dac8e680be12d11ebda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:07 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:46990
I20260926 07:28:07.866148 10781 raft_consensus.cc:2229] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.866271 10782 tablet_replica.cc:330] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: stopping tablet replica
I20260926 07:28:07.866335 10781 raft_consensus.cc:2258] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.866446 10782 raft_consensus.cc:2229] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:07.866604 10780 tablet_replica.cc:330] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: stopping tablet replica
I20260926 07:28:07.866653 10782 raft_consensus.cc:2258] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.866765 10780 raft_consensus.cc:2229] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:07.867226 10780 raft_consensus.cc:2258] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:07.868381 10782 ts_tablet_manager.cc:1905] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.868381 10781 ts_tablet_manager.cc:1905] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.871675 10209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:07.872474 10209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:07.873407 10780 ts_tablet_manager.cc:1905] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:07.876268 10782 ts_tablet_manager.cc:1918] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.876456 10782 log.cc:1196] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/7625bf8c0bf64dac8e680be12d11ebda
I20260926 07:28:07.876878 10782 ts_tablet_manager.cc:1939] T 7625bf8c0bf64dac8e680be12d11ebda P 72530ca0003341fe8720fa5e2cc9295f: Deleting consensus metadata
I20260926 07:28:07.878031 10197 catalog_manager.cc:4841] TS 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221): tablet 7625bf8c0bf64dac8e680be12d11ebda (table test-table [id=23773d644816488092ee3fee14b16a62]) successfully deleted
I20260926 07:28:07.880224 10487 tablet_service.cc:1467] Processing CreateTablet for tablet b7a14e9b458149bc9f90471a83046e1c (DEFAULT_TABLE table=test-table [id=0faacaa96a2646c5bd59bea47d960e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.880332 10626 tablet_service.cc:1467] Processing CreateTablet for tablet b7a14e9b458149bc9f90471a83046e1c (DEFAULT_TABLE table=test-table [id=0faacaa96a2646c5bd59bea47d960e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.880584 10487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a14e9b458149bc9f90471a83046e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.880740 10626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a14e9b458149bc9f90471a83046e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.880980 10347 tablet_service.cc:1467] Processing CreateTablet for tablet b7a14e9b458149bc9f90471a83046e1c (DEFAULT_TABLE table=test-table [id=0faacaa96a2646c5bd59bea47d960e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:07.881371 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a14e9b458149bc9f90471a83046e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:07.882926 10780 ts_tablet_manager.cc:1918] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.883113 10780 log.cc:1196] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/7625bf8c0bf64dac8e680be12d11ebda
I20260926 07:28:07.883605 10780 ts_tablet_manager.cc:1939] T 7625bf8c0bf64dac8e680be12d11ebda P dfcdd42b7535473982fa5b3dd3d449d6: Deleting consensus metadata
I20260926 07:28:07.885841 10195 catalog_manager.cc:4841] TS dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763): tablet 7625bf8c0bf64dac8e680be12d11ebda (table test-table [id=23773d644816488092ee3fee14b16a62]) successfully deleted
I20260926 07:28:07.886672 10722 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Bootstrap starting.
I20260926 07:28:07.886804 10725 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Bootstrap starting.
I20260926 07:28:07.888063 10781 ts_tablet_manager.cc:1918] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:07.888222 10781 log.cc:1196] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/7625bf8c0bf64dac8e680be12d11ebda
I20260926 07:28:07.888367 10725 tablet_bootstrap.cc:654] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.888672 10781 ts_tablet_manager.cc:1939] T 7625bf8c0bf64dac8e680be12d11ebda P 60dc28a1550b4218be2df6b4a91246e9: Deleting consensus metadata
I20260926 07:28:07.888741 10722 tablet_bootstrap.cc:654] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.889777 10198 catalog_manager.cc:4841] TS 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703): tablet 7625bf8c0bf64dac8e680be12d11ebda (table test-table [id=23773d644816488092ee3fee14b16a62]) successfully deleted
I20260926 07:28:07.890852 10721 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Bootstrap starting.
I20260926 07:28:07.891598 10725 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: No bootstrap required, opened a new log
I20260926 07:28:07.891757 10725 ts_tablet_manager.cc:1397] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:28:07.892356 10721 tablet_bootstrap.cc:654] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:07.892427 10725 raft_consensus.cc:348] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.892655 10725 raft_consensus.cc:374] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.892729 10725 raft_consensus.cc:729] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72530ca0003341fe8720fa5e2cc9295f, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.892930 10725 consensus_queue.cc:260] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.893450 10725 ts_tablet_manager.cc:1428] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:07.893914 10722 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: No bootstrap required, opened a new log
I20260926 07:28:07.894080 10722 ts_tablet_manager.cc:1397] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 07:28:07.894753 10722 raft_consensus.cc:348] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.895020 10722 raft_consensus.cc:374] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.895112 10722 raft_consensus.cc:729] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.895398 10722 consensus_queue.cc:260] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.895982 10722 ts_tablet_manager.cc:1428] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:07.897289 10721 tablet_bootstrap.cc:492] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: No bootstrap required, opened a new log
I20260926 07:28:07.897445 10721 ts_tablet_manager.cc:1397] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 07:28:07.898172 10721 raft_consensus.cc:348] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.898434 10721 raft_consensus.cc:374] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:07.898528 10721 raft_consensus.cc:729] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcdd42b7535473982fa5b3dd3d449d6, State: Initialized, Role: FOLLOWER
I20260926 07:28:07.898977 10721 consensus_queue.cc:260] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:07.899883 10721 ts_tablet_manager.cc:1428] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:28:08.161504 10727 raft_consensus.cc:482] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:08.161722 10727 raft_consensus.cc:504] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:08.163424 10727 leader_election.cc:288] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:08.173935 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.164033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:44213 (local address 127.2.147.193:45763)
0926 07:28:08.164233 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.164238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.164260 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:28:08.164508 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.164517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.164629 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.164700 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:08.165525 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.166957 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.170489 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.172275 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.172464 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.172620 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.173338 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.173531 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:28:08.174877 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.163605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.2.147.194:44213)
0926 07:28:08.164134 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:08.164157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:08.164310 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:08.164937 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:08.164945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:08.164987 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:08.165331 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.165345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.167187 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.167194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.170276 (+ 3082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.170295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.172852 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.172860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.173119 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.173170 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:28:08.174363 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0926 07:28:08.174502 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":659,"threads_started":1}
I20260926 07:28:08.175490 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a14e9b458149bc9f90471a83046e1c" candidate_uuid: "60dc28a1550b4218be2df6b4a91246e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" is_pre_election: true
I20260926 07:28:08.175779 10373 raft_consensus.cc:2454] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60dc28a1550b4218be2df6b4a91246e9 in term 0.
I20260926 07:28:08.176304 10438 leader_election.cc:302] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:08.176703 10727 raft_consensus.cc:2781] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:08.176821 10727 raft_consensus.cc:482] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:08.176904 10727 raft_consensus.cc:3037] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:08.179955 10727 raft_consensus.cc:504] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:08.180538 10727 leader_election.cc:288] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [CANDIDATE]: Term 1 election: Requested vote from peers dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763), 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221)
I20260926 07:28:08.180960 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a14e9b458149bc9f90471a83046e1c" candidate_uuid: "60dc28a1550b4218be2df6b4a91246e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcdd42b7535473982fa5b3dd3d449d6"
I20260926 07:28:08.181170 10373 raft_consensus.cc:3037] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:08.181687 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.163750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:33221 (local address 127.2.147.194:44503)
0926 07:28:08.174556 (+ 10806us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:08.174577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:08.174722 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:08.175244 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:08.175249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:08.175273 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:08.175598 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.175626 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.177061 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.177065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.179197 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.179231 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.180871 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.180877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.181067 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.181099 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:28:08.181187 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:28:08.181319 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10683,"thread_start_us":10525,"threads_started":1}
I20260926 07:28:08.181706 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.163973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:44503 (local address 127.2.147.195:33221)
0926 07:28:08.174283 (+ 10310us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.174288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.174679 (+ 391us) server_negotiation.cc:407] Connection header received
0926 07:28:08.174857 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.174865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.174952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.175058 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:08.175830 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.176905 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.179357 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.180485 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.180658 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.180731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.181237 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.181436 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10188,"thread_start_us":10086,"threads_started":1}
I20260926 07:28:08.182698 10646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a14e9b458149bc9f90471a83046e1c" candidate_uuid: "60dc28a1550b4218be2df6b4a91246e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f" is_pre_election: true
I20260926 07:28:08.182914 10646 raft_consensus.cc:2454] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60dc28a1550b4218be2df6b4a91246e9 in term 0.
I20260926 07:28:08.183259 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a14e9b458149bc9f90471a83046e1c" candidate_uuid: "60dc28a1550b4218be2df6b4a91246e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72530ca0003341fe8720fa5e2cc9295f"
I20260926 07:28:08.183421 10644 raft_consensus.cc:3037] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:08.184118 10373 raft_consensus.cc:2454] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60dc28a1550b4218be2df6b4a91246e9 in term 1.
I20260926 07:28:08.185607 10644 raft_consensus.cc:2454] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60dc28a1550b4218be2df6b4a91246e9 in term 1.
I20260926 07:28:08.185865 10438 leader_election.cc:302] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60dc28a1550b4218be2df6b4a91246e9, dfcdd42b7535473982fa5b3dd3d449d6; no voters:
I20260926 07:28:08.186156 10727 raft_consensus.cc:2781] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:08.187094 10727 raft_consensus.cc:686] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 1 LEADER]: Becoming Leader. State: Replica: 60dc28a1550b4218be2df6b4a91246e9, State: Running, Role: LEADER
I20260926 07:28:08.187533 10727 consensus_queue.cc:237] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } }
I20260926 07:28:08.191041 10208 catalog_manager.cc:5495] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "60dc28a1550b4218be2df6b4a91246e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60dc28a1550b4218be2df6b4a91246e9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20260926 07:28:08.276860 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.267263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.2.147.194:34703)
0926 07:28:08.271645 (+ 4382us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.271648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.271666 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:28:08.271739 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.271742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.271811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.271977 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:08.272394 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.273215 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.274593 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.275010 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.275144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.275624 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:08.276207 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:08.276362 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.276413 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.276494 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4292,"thread_start_us":4170,"threads_started":1}
I20260926 07:28:08.290961 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.285957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:38875 (local address 127.2.147.193:45763)
0926 07:28:08.286108 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.286112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.286715 (+ 603us) server_negotiation.cc:407] Connection header received
0926 07:28:08.286844 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.286848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.286907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.286978 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:08.287469 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.288272 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.289616 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.290001 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.290088 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.290422 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:08.290524 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:08.290586 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.290698 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.290756 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:28:08.291020 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.285954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45763 (local address 127.2.147.194:38875)
0926 07:28:08.286632 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:08.286648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:08.286750 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:08.287001 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:08.287004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:08.287012 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:28:08.287342 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.287350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.288382 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.288386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.289508 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.289519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.290144 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.290148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.290246 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.290289 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:28:08.290604 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:28:08.290611 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:28:08.290666 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:28:08.290758 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":351,"threads_started":1}
I20260926 07:28:08.298132 10644 raft_consensus.cc:1261] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Refusing update from remote peer 60dc28a1550b4218be2df6b4a91246e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:08.298171 10373 raft_consensus.cc:1261] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Refusing update from remote peer 60dc28a1550b4218be2df6b4a91246e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:08.298719 10808 consensus_queue.cc:1035] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72530ca0003341fe8720fa5e2cc9295f" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:08.299039 10727 consensus_queue.cc:1035] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcdd42b7535473982fa5b3dd3d449d6" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:08.332602 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.321363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:42837 (local address 127.2.147.194:34703)
0926 07:28:08.321550 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.321556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.327680 (+ 6124us) server_negotiation.cc:407] Connection header received
0926 07:28:08.327868 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.327873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.327941 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.328066 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:08.328517 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.329254 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.330910 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.331396 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.331532 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.331902 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:08.331989 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:08.332086 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.332231 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.332301 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:28:08.332700 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.321263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:34703 (local address 127.2.147.193:42837)
0926 07:28:08.327586 (+ 6323us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:08.327601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:08.327742 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:08.328083 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:08.328088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:08.328099 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:28:08.328366 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.328375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.329381 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.329388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.330770 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:08.330781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.331565 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.331571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:08.331702 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.331741 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:28:08.332095 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:28:08.332106 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:28:08.332195 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:28:08.332317 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6196,"thread_start_us":6094,"threads_started":1}
I20260926 07:28:08.361722 10208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:08.361974 10208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:08.364926 10208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 062e1698d18245cdb7b5f23af770c515: Sending DeleteTablet for 3 replicas of tablet b7a14e9b458149bc9f90471a83046e1c
I20260926 07:28:08.365649 10347 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a14e9b458149bc9f90471a83046e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:08 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53914
I20260926 07:28:08.365656 10487 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a14e9b458149bc9f90471a83046e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:08 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44208
I20260926 07:28:08.365928 10780 tablet_replica.cc:330] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: stopping tablet replica
I20260926 07:28:08.365976 10781 tablet_replica.cc:330] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: stopping tablet replica
I20260926 07:28:08.366093 10780 raft_consensus.cc:2229] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:08.366154 10781 raft_consensus.cc:2229] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:08.366283 10780 raft_consensus.cc:2258] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:08.366663 10781 raft_consensus.cc:2258] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:08.366753 10626 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a14e9b458149bc9f90471a83046e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:08 UTC) from {username='oryx', principal='oryx/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:46990
I20260926 07:28:08.367296 10782 tablet_replica.cc:330] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: stopping tablet replica
I20260926 07:28:08.367472 10782 raft_consensus.cc:2229] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:08.367580 10780 ts_tablet_manager.cc:1905] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:08.367663 10782 raft_consensus.cc:2258] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:08.368043 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10282
I20260926 07:28:08.368790 10781 ts_tablet_manager.cc:1905] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:08.375327 10780 ts_tablet_manager.cc:1918] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:08.375466 10780 log.cc:1196] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/b7a14e9b458149bc9f90471a83046e1c
I20260926 07:28:08.375794 10781 ts_tablet_manager.cc:1918] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:08.375924 10781 log.cc:1196] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/b7a14e9b458149bc9f90471a83046e1c
I20260926 07:28:08.376008 10780 ts_tablet_manager.cc:1939] T b7a14e9b458149bc9f90471a83046e1c P dfcdd42b7535473982fa5b3dd3d449d6: Deleting consensus metadata
I20260926 07:28:08.376376 10781 ts_tablet_manager.cc:1939] T b7a14e9b458149bc9f90471a83046e1c P 60dc28a1550b4218be2df6b4a91246e9: Deleting consensus metadata
I20260926 07:28:08.377185 10195 catalog_manager.cc:4841] TS dfcdd42b7535473982fa5b3dd3d449d6 (127.2.147.193:45763): tablet b7a14e9b458149bc9f90471a83046e1c (table test-table [id=0faacaa96a2646c5bd59bea47d960e62]) successfully deleted
I20260926 07:28:08.377360 10198 catalog_manager.cc:4841] TS 60dc28a1550b4218be2df6b4a91246e9 (127.2.147.194:34703): tablet b7a14e9b458149bc9f90471a83046e1c (table test-table [id=0faacaa96a2646c5bd59bea47d960e62]) successfully deleted
I20260926 07:28:08.378165 10782 ts_tablet_manager.cc:1905] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:08.383255 10782 ts_tablet_manager.cc:1918] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:28:08.383395 10782 log.cc:1196] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/b7a14e9b458149bc9f90471a83046e1c
I20260926 07:28:08.383809 10782 ts_tablet_manager.cc:1939] T b7a14e9b458149bc9f90471a83046e1c P 72530ca0003341fe8720fa5e2cc9295f: Deleting consensus metadata
I20260926 07:28:08.384836 10197 catalog_manager.cc:4841] TS 72530ca0003341fe8720fa5e2cc9295f (127.2.147.195:33221): tablet b7a14e9b458149bc9f90471a83046e1c (table test-table [id=0faacaa96a2646c5bd59bea47d960e62]) successfully deleted
Sep 26 07:28:08 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.193@KRBTEST.COM
I20260926 07:28:08.393996 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.369579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.2.147.193:45763)
0926 07:28:08.369738 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.369744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.378124 (+ 8380us) server_negotiation.cc:407] Connection header received
0926 07:28:08.378297 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.378302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.378385 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.378492 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:08.379997 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.381050 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.382842 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.383523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.383665 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.388705 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:08.388719 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:08.388733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:08.388767 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:08.391490 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.392104 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.392111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.392113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.392230 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.392622 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.392626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.392628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.392940 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:08.393250 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.393505 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.393730 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:28:08.394625 10413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:08.573869 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10282
I20260926 07:28:08.599881 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10422
Sep 26 07:28:08 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.194@KRBTEST.COM
I20260926 07:28:08.615363 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.600612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.2.147.194:34703)
0926 07:28:08.600819 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.600826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.600870 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:28:08.601102 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.601106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.601164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.601245 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:08.602871 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.603672 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.605061 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.605493 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.605631 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.609638 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:08.609648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:08.609675 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:08.609710 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:08.612654 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.613272 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.613279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.613281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.613409 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.613805 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.613809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.613811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.614111 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:08.614464 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.614636 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.614835 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:28:08.615943 10552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:08.741454 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10422
I20260926 07:28:08.762440 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10561
Sep 26 07:28:08 dist-test-slave-1dwd krb5kdc[10166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.147.195@KRBTEST.COM
I20260926 07:28:08.782871 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:08.763225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.2.147.195:33221)
0926 07:28:08.763703 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:08.763706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:08.763724 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:28:08.763774 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:08.763777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:08.763832 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:08.763920 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:08.765068 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.765775 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.766937 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:08.767406 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:08.767492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:08.776997 (+ 9505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:08.777010 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:08.777026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:08.777061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:08.780276 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.780882 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.780895 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.780898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.781023 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:08.781395 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:08.781402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:08.781405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:08.781739 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:08.782044 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:08.782290 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:28:08.782529 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":279,"threads_started":1}
I20260926 07:28:08.783499 10691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:08.903141 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10561
I20260926 07:28:08.923775 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10179
I20260926 07:28:08.924896 10251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:09.035107 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10179
2026-09-26T07:28:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6901 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:09 dist-test-slave-1dwd krb5kdc[10829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:09 dist-test-slave-1dwd krb5kdc[10829](info): set up 2 sockets
Sep 26 07:28:09 dist-test-slave-1dwd krb5kdc[10829](info): commencing operation
krb5kdc: starting...
W20260926 07:28:10.116217 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:28:10 dist-test-slave-1dwd krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:28:10.317929 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:44091
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:10.629298 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:10.629563 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:10.629647 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:10.636647 10842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:10.636740 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:10.636785 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:10.636839 10842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:10.636879 10842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:10.646306 10842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:10.647509 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:10.648756 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:10.660820 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:10.660908 10842 server_base.cc:1034] running on GCE node
W20260926 07:28:10.660768 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:10.660806 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:10.661933 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:10.663285 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:10.664491 10842 hybrid_clock.cc:648] HybridClock initialized: now 1790407690664450 us; error 35 us; skew 500 ppm
I20260926 07:28:10.664793 10842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:28:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1696 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:10 dist-test-slave-1dwd krb5kdc[10856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:10 dist-test-slave-1dwd krb5kdc[10856](info): set up 2 sockets
Sep 26 07:28:10 dist-test-slave-1dwd krb5kdc[10856](info): commencing operation
krb5kdc: starting...
W20260926 07:28:11.804076 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:28:11 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:28:12.002641 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34941
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:12.312430 10869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:12.312775 10869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:12.312942 10869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:12.319895 10869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:12.320024 10869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:12.320123 10869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:12.320222 10869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:12.320304 10869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:12.329578 10869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:12.330791 10869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:12.332059 10869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:12.343154 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:12.343179 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:12.343252 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:12.344193 10869 server_base.cc:1034] running on GCE node
I20260926 07:28:12.344925 10869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:12.346226 10869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:12.347466 10869 hybrid_clock.cc:648] HybridClock initialized: now 1790407692347408 us; error 45 us; skew 500 ppm
I20260926 07:28:12.347787 10869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:12 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407692, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:12.350741 10869 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:28:12.352190 10869 webserver.cc:460] Webserver started at http://127.2.147.254:44061/ using document root <none> and password file <none>
I20260926 07:28:12.352859 10869 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:12.353003 10869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:12.353302 10869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:12.356264 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "56bc09d1392346fbae0fa0be0c287c6a"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "7c23dbefd6de4e11b7fb9c273795d565"
server_key_iv: "114290d29f6861aeb51531b1940b3af3"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.357023 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "56bc09d1392346fbae0fa0be0c287c6a"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "7c23dbefd6de4e11b7fb9c273795d565"
server_key_iv: "114290d29f6861aeb51531b1940b3af3"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.361891 10869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:28:12.365144 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:12.366362 10869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:28:12.366575 10869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "56bc09d1392346fbae0fa0be0c287c6a"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "7c23dbefd6de4e11b7fb9c273795d565"
server_key_iv: "114290d29f6861aeb51531b1940b3af3"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.366758 10869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:12.385324 10869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:12.388943 10869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:12.389207 10869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:12.406949 10869 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34941
I20260926 07:28:12.406978 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34941 every 8 connection(s)
I20260926 07:28:12.408214 10869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:12.409868 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10869
I20260926 07:28:12.410070 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:28:12.410667 2639 external_mini_cluster.cc:1428] Setting key 5609f1c5fcf4643b9dd1b60d1dbfff4f
I20260926 07:28:12.412176 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:28:12 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:12.425119 10936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Bootstrap starting.
I20260926 07:28:12.429028 10936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:12.430222 10936 log.cc:824] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:12.433405 10936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: No bootstrap required, opened a new log
I20260926 07:28:12.433738 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:12.412545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.2.147.254:34941)
0926 07:28:12.413332 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:12.413349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:12.413398 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:28:12.415377 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:12.415403 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:12.415870 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:12.416283 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:12.417501 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:12.419544 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:12.420746 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:12.421715 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:12.421894 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:12.425759 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:12.425769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:12.425786 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:12.425827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:12.429624 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:12.430215 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:12.430224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:12.430230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:12.430421 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:12.430808 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:12.430819 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:12.430825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:12.431449 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:12.431730 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:12.432086 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 07:28:12.432478 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":313,"threads_started":1}
I20260926 07:28:12.441295 10936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:12.441579 10936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:12.441648 10936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bc09d1392346fbae0fa0be0c287c6a, State: Initialized, Role: FOLLOWER
I20260926 07:28:12.442071 10936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:12.442250 10936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:12.442334 10936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:12.442448 10936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:12.444344 10936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:12.444741 10936 leader_election.cc:302] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56bc09d1392346fbae0fa0be0c287c6a; no voters:
I20260926 07:28:12.445385 10936 leader_election.cc:288] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:12.445479 10941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:12.446172 10941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 LEADER]: Becoming Leader. State: Replica: 56bc09d1392346fbae0fa0be0c287c6a, State: Running, Role: LEADER
I20260926 07:28:12.446717 10941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:12.447376 10936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:12.449280 10942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56bc09d1392346fbae0fa0be0c287c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } } }
I20260926 07:28:12.449304 10943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56bc09d1392346fbae0fa0be0c287c6a. Latest consensus state: current_term: 1 leader_uuid: "56bc09d1392346fbae0fa0be0c287c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } } }
I20260926 07:28:12.449694 10943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:12.449694 10942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:12.453438 10946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:12.458990 10946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:12.467384 10946 catalog_manager.cc:1347] Generated new cluster ID: 29b801a5fb444e46b4811368852c4371
I20260926 07:28:12.467514 10946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:12.488245 10946 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:12.489522 10946 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:12.496519 10946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Generated new TSK 0
I20260926 07:28:12.497287 10946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:28:12.552764 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:12.891510 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:12.891784 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:12.891948 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:12.899109 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:12.899258 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:12.899448 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:12.908926 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:12.910169 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:12.911614 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:12.927708 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:12.927802 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:12.927670 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:12.928529 10962 server_base.cc:1034] running on GCE node
I20260926 07:28:12.929108 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:12.930016 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:12.931288 10962 hybrid_clock.cc:648] HybridClock initialized: now 1790407692931243 us; error 46 us; skew 500 ppm
I20260926 07:28:12.931599 10962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:12 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407692, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:12.934741 10962 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:28:12.936112 10962 webserver.cc:460] Webserver started at http://127.2.147.193:34173/ using document root <none> and password file <none>
I20260926 07:28:12.936688 10962 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:12.936781 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:12.937021 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:12.939783 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "b8119c1032d3460ba6cd8e681a6de678"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "83856a4a586d60fae7939c3789c1dc0c"
server_key_iv: "fc82f8eeb0b0a21447606f34c2c1d5e7"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.940467 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "b8119c1032d3460ba6cd8e681a6de678"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "83856a4a586d60fae7939c3789c1dc0c"
server_key_iv: "fc82f8eeb0b0a21447606f34c2c1d5e7"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.945204 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:28:12.948520 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:12.949815 10962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:12.949959 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b8119c1032d3460ba6cd8e681a6de678"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "83856a4a586d60fae7939c3789c1dc0c"
server_key_iv: "fc82f8eeb0b0a21447606f34c2c1d5e7"
server_key_version: "encryptionkey@0"
I20260926 07:28:12.950165 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:12.970005 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:12.973791 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:12.974001 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:12.974792 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:12.976186 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:12.976254 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:12.976334 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:12.976375 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.005971 10962 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45255
I20260926 07:28:13.006045 11089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45255 every 8 connection(s)
I20260926 07:28:13.007184 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:13.014542 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 10962
I20260926 07:28:13.014725 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:13.015398 2639 external_mini_cluster.cc:1428] Setting key a9af406072474ad0cdb9b61da3ebf626
Sep 26 07:28:13 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407692, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:13.026383 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.009701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:44899 (local address 127.2.147.254:34941)
0926 07:28:13.010154 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:13.010158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:13.010956 (+ 798us) server_negotiation.cc:407] Connection header received
0926 07:28:13.012760 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:13.012766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:13.012851 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:13.012968 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:13.014362 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.015151 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.016842 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.017237 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.017364 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:13.020856 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:13.020861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:13.020865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:13.020894 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:13.023428 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:13.024186 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:13.024192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:13.024196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:13.024352 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:13.024837 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:13.024841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:13.024843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:13.025081 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:13.025187 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:13.025993 (+ 806us) server_negotiation.cc:299] Negotiation successful
0926 07:28:13.026144 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":263,"threads_started":1}
I20260926 07:28:13.027815 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.010160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:44899)
0926 07:28:13.010803 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:13.010844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:13.012494 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:13.013197 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:13.013205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:13.013669 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:13.014195 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.014212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.015312 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.015318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.016708 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.016718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.017460 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.017469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.017684 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:13.018342 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:13.018366 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:13.020629 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:13.023682 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:13.023688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:13.023699 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:13.024071 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:13.024459 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:13.024462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:13.024465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:13.024732 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:13.025201 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:13.025211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:13.025614 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 07:28:13.025923 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 07:28:13.026436 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260926 07:28:13.029261 11090 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34941
I20260926 07:28:13.029534 11090 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:13.030194 11090 heartbeater.cc:511] Master 127.2.147.254:34941 requested a full tablet report, sending...
I20260926 07:28:13.032119 10901 ts_manager.cc:194] Registered new tserver with Master: b8119c1032d3460ba6cd8e681a6de678 (127.2.147.193:45255)
I20260926 07:28:13.033605 10901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:44899
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:28:13.064230 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:13.373569 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:13.373865 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:13.374025 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:13.380926 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:13.381058 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:13.381232 11096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:13.390316 11096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:13.391515 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:13.392772 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:13.405421 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:13.405437 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:13.405640 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:13.406049 11096 server_base.cc:1034] running on GCE node
I20260926 07:28:13.406621 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:13.407545 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:13.408758 11096 hybrid_clock.cc:648] HybridClock initialized: now 1790407693408702 us; error 49 us; skew 500 ppm
I20260926 07:28:13.409049 11096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:13 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:13.411998 11096 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:28:13.413403 11096 webserver.cc:460] Webserver started at http://127.2.147.194:36809/ using document root <none> and password file <none>
I20260926 07:28:13.414032 11096 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:13.414156 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:13.414454 11096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:13.417183 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "087371a85a04433d9730b03784de5454"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "d58af5f2e94a4a7f8520bd5f389838d0"
server_key_iv: "d9c76d6904ff023e0e85206de76e842a"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.417927 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "087371a85a04433d9730b03784de5454"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "d58af5f2e94a4a7f8520bd5f389838d0"
server_key_iv: "d9c76d6904ff023e0e85206de76e842a"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.422540 11096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:28:13.425835 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.426988 11096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:13.427143 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "087371a85a04433d9730b03784de5454"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "d58af5f2e94a4a7f8520bd5f389838d0"
server_key_iv: "d9c76d6904ff023e0e85206de76e842a"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.427372 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:13.446494 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:13.450598 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:13.450882 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:13.451772 11096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:13.453126 11096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:13.453197 11096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.453302 11096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:13.453383 11096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.486349 11096 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:40913
I20260926 07:28:13.486392 11223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:40913 every 8 connection(s)
I20260926 07:28:13.487627 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:13.492010 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11096
I20260926 07:28:13.492182 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:13.492625 2639 external_mini_cluster.cc:1428] Setting key ffa0dfd8c3606055af0a977512b212fa
Sep 26 07:28:13 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:13.506721 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.490025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:54251 (local address 127.2.147.254:34941)
0926 07:28:13.490192 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:13.490196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:13.491387 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 07:28:13.493305 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:13.493310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:13.493418 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:13.493531 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:13.494946 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.495687 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.497379 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.497762 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.497888 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:13.501436 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:13.501441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:13.501444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:13.501474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:13.503929 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:13.504604 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:13.504611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:13.504614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:13.504743 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:13.505132 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:13.505137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:13.505141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:13.505386 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:13.505528 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:13.506175 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 07:28:13.506383 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:28:13.507745 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.490530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:54251)
0926 07:28:13.491191 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:13.491286 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:13.493057 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:13.493758 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:13.493766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:13.494245 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:13.494779 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.494795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.495833 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.495838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.497238 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.497249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.497969 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.497975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.498193 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:13.498895 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:13.498920 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:13.501229 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:13.504064 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:13.504072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:13.504087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:13.504487 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:13.504850 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:13.504855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:13.504858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:13.505025 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:13.505516 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:13.505524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:13.505826 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:28:13.506095 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 07:28:13.506593 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260926 07:28:13.509181 11224 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34941
I20260926 07:28:13.509459 11224 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:13.510113 11224 heartbeater.cc:511] Master 127.2.147.254:34941 requested a full tablet report, sending...
I20260926 07:28:13.511549 10901 ts_manager.cc:194] Registered new tserver with Master: 087371a85a04433d9730b03784de5454 (127.2.147.194:40913)
I20260926 07:28:13.512341 10901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:54251
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:28:13.538235 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:13.856669 11229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:13.856951 11229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:13.857043 11229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:13.864041 11229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:13.864140 11229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:13.864248 11229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:13.873878 11229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:13.875108 11229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:13.876475 11229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:13.889667 11237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:13.889730 11235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:13.889631 11234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:13.890453 11229 server_base.cc:1034] running on GCE node
I20260926 07:28:13.891075 11229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:13.892009 11229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:13.893230 11229 hybrid_clock.cc:648] HybridClock initialized: now 1790407693893194 us; error 49 us; skew 500 ppm
I20260926 07:28:13.893612 11229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:13 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:13.900014 11229 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:28:13.901397 11229 webserver.cc:460] Webserver started at http://127.2.147.195:44001/ using document root <none> and password file <none>
I20260926 07:28:13.902006 11229 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:13.902117 11229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:13.902391 11229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:13.905202 11229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "78ea6554fc7b4b98a63e04a57c8aa235"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "26f7f76200f72c69b178ef74f30b0ecc"
server_key_iv: "a2e77908c48c6d2db71daf1ecb943dd4"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.905901 11229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "78ea6554fc7b4b98a63e04a57c8aa235"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "26f7f76200f72c69b178ef74f30b0ecc"
server_key_iv: "a2e77908c48c6d2db71daf1ecb943dd4"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.910645 11229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 07:28:13.913932 11244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.915128 11229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:28:13.915325 11229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "78ea6554fc7b4b98a63e04a57c8aa235"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "26f7f76200f72c69b178ef74f30b0ecc"
server_key_iv: "a2e77908c48c6d2db71daf1ecb943dd4"
server_key_version: "encryptionkey@0"
I20260926 07:28:13.915515 11229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:13.947412 11229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:13.951411 11229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:13.951705 11229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:13.952646 11229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:13.954123 11229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:13.954221 11229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.954350 11229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:13.954423 11229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:13.986821 11229 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:43059
I20260926 07:28:13.986842 11356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:43059 every 8 connection(s)
I20260926 07:28:13.988117 11229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:13.997087 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11229
I20260926 07:28:13.997277 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:13.997715 2639 external_mini_cluster.cc:1428] Setting key 0cdddd482add06439b52c55ed92124e6
Sep 26 07:28:13 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:14.007534 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.990633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:47613 (local address 127.2.147.254:34941)
0926 07:28:13.990805 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:13.990809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:13.991916 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 07:28:13.993684 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:13.993690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:13.993807 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:13.993915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:13.995371 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.996089 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.997751 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.998257 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.998427 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:14.002241 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:14.002246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:14.002249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:14.002278 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:14.004642 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.005293 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.005300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.005308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.005430 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.005859 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.005863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.005868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.006122 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:14.006272 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:14.006941 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 07:28:14.007164 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:28:14.008594 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:13.991047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:47613)
0926 07:28:13.991763 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:13.991800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:13.993410 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:13.994144 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:13.994155 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:13.994631 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:13.995162 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.995177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.996247 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.996252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.997601 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:13.997613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:13.998520 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:13.998529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:13.998817 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:13.999581 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:13.999636 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:14.002070 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:14.004754 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:14.004761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:14.004772 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:14.005165 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:14.005554 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:14.005560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:14.005563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:14.005743 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:14.006257 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:14.006264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:14.006574 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:28:14.006852 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 07:28:14.007396 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
I20260926 07:28:14.009974 11357 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34941
I20260926 07:28:14.010260 11357 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:14.010913 11357 heartbeater.cc:511] Master 127.2.147.254:34941 requested a full tablet report, sending...
I20260926 07:28:14.012491 10901 ts_manager.cc:194] Registered new tserver with Master: 78ea6554fc7b4b98a63e04a57c8aa235 (127.2.147.195:43059)
I20260926 07:28:14.013170 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:28:14.013470 10901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:47613
WARNING: no policy specified for oryx/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.193@KRBTEST.COM" created.
I20260926 07:28:14.037125 11090 heartbeater.cc:503] Master 127.2.147.254:34941 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
Entry for principal oryx/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab.
WARNING: no policy specified for oryx/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
Entry for principal oryx/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab.
WARNING: no policy specified for oryx/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
Entry for principal oryx/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab.
I20260926 07:28:14.148123 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10962
Sep 26 07:28:14 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:28:14.163774 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:14.149176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.2.147.193:45255)
0926 07:28:14.149798 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:14.149804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:14.149832 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:28:14.149895 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:14.149902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:14.150135 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:14.150282 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:14.151674 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.152673 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.154077 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.154620 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.154757 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:14.158670 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:14.158680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:14.158694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:14.158728 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:14.161434 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.162032 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.162039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.162041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.162151 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.162437 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.162440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.162441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.162731 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:14.162933 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:14.163151 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:28:14.163387 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":302,"threads_started":1}
I20260926 07:28:14.164966 11085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:14.246049 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10962
I20260926 07:28:14.262357 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11096
Sep 26 07:28:14 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:28:14.279649 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:14.263260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.2.147.194:40913)
0926 07:28:14.263764 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:14.263771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:14.263794 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:28:14.263851 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:14.263857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:14.264057 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:14.264177 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:14.265314 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.266239 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.267469 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.267980 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.268073 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:14.272105 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:14.272115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:14.272128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:14.272183 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:14.275941 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.276783 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.276792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.276795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.276934 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.277328 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.277332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.277336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.277754 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:14.278913 (+ 1159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:14.279089 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:28:14.279362 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260926 07:28:14.280956 11219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:14.361282 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11096
I20260926 07:28:14.376855 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11229
Sep 26 07:28:14 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:28:14.393242 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:14.377827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.2.147.195:43059)
0926 07:28:14.378414 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:14.378421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:14.378449 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:28:14.378529 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:14.378536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:14.378750 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:14.378928 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:14.380283 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.381263 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.382507 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:14.383058 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:14.383194 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:14.387520 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:14.387530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:14.387544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:14.387575 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:14.390399 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.391036 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.391043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.391046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.391171 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:14.391723 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:14.391729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:14.391733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:14.392053 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:14.392375 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:14.392668 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 07:28:14.392919 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
I20260926 07:28:14.394878 11352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:14.475790 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11229
I20260926 07:28:14.492987 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 10869
I20260926 07:28:14.494144 10931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:14.595319 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 10869
I20260926 07:28:14.615365 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34941
--webserver_interface=127.2.147.254
--webserver_port=44061
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:14.927404 11374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:14.927738 11374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:14.927901 11374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:14.934875 11374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:14.935015 11374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:14.935144 11374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:14.935279 11374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:14.935369 11374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:14.944674 11374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=44061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:14.945957 11374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:14.947250 11374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:14.958529 11382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:14.958691 11379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:14.958537 11380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:14.959403 11374 server_base.cc:1034] running on GCE node
I20260926 07:28:14.960187 11374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:14.961633 11374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:14.962847 11374 hybrid_clock.cc:648] HybridClock initialized: now 1790407694962788 us; error 34 us; skew 500 ppm
I20260926 07:28:14.963188 11374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:14 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:14.966125 11374 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:28:14.967443 11374 webserver.cc:460] Webserver started at http://127.2.147.254:44061/ using document root <none> and password file <none>
I20260926 07:28:14.968063 11374 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:14.968165 11374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:14.973145 11374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:28:14.975765 11389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:14.977077 11374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:14.977265 11374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "56bc09d1392346fbae0fa0be0c287c6a"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "7c23dbefd6de4e11b7fb9c273795d565"
server_key_iv: "114290d29f6861aeb51531b1940b3af3"
server_key_version: "encryptionkey@0"
I20260926 07:28:14.978188 11374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:14.994267 11374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:14.998234 11374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:14.998498 11374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:15.016168 11374 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34941
I20260926 07:28:15.016194 11440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34941 every 8 connection(s)
I20260926 07:28:15.017539 11374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:15.023952 11441 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:28:15.023944 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11374
I20260926 07:28:15.024708 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:45255
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:34941
--webserver_port=34173
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:15.026995 11441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Bootstrap starting.
I20260926 07:28:15.045070 11441 log.cc:824] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:15.053530 11441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:28:15.054033 11441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Bootstrap complete.
I20260926 07:28:15.062800 11441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:15.063697 11441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bc09d1392346fbae0fa0be0c287c6a, State: Initialized, Role: FOLLOWER
I20260926 07:28:15.064327 11441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:15.064577 11441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:15.064703 11441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:15.064873 11441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:28:15.068310 11441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:15.068778 11441 leader_election.cc:302] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56bc09d1392346fbae0fa0be0c287c6a; no voters:
I20260926 07:28:15.069401 11441 leader_election.cc:288] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:28:15.069605 11445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:28:15.071029 11445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [term 2 LEADER]: Becoming Leader. State: Replica: 56bc09d1392346fbae0fa0be0c287c6a, State: Running, Role: LEADER
I20260926 07:28:15.071762 11441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:15.071648 11445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } }
I20260926 07:28:15.074106 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56bc09d1392346fbae0fa0be0c287c6a. Latest consensus state: current_term: 2 leader_uuid: "56bc09d1392346fbae0fa0be0c287c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } } }
I20260926 07:28:15.074523 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:15.078939 11451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:15.078665 11446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56bc09d1392346fbae0fa0be0c287c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bc09d1392346fbae0fa0be0c287c6a" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34941 } } }
I20260926 07:28:15.079107 11446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:15.084864 11451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:15.086197 11451 catalog_manager.cc:1259] Loaded cluster ID: 29b801a5fb444e46b4811368852c4371
I20260926 07:28:15.086337 11451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:15.090129 11451 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:15.091614 11451 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56bc09d1392346fbae0fa0be0c287c6a: Loaded TSK: 0
I20260926 07:28:15.093182 11451 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:28:15.351188 11443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:15.351562 11443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:15.351717 11443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:15.358978 11443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:15.359123 11443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:15.359315 11443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:15.368865 11443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.193.keytab
--principal=oryx/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:45255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=34173
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:15.370129 11443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:15.371471 11443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:15.385703 11468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:15.386133 11470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:15.386399 11467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:15.387137 11443 server_base.cc:1034] running on GCE node
I20260926 07:28:15.387840 11443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:15.388979 11443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:15.390223 11443 hybrid_clock.cc:648] HybridClock initialized: now 1790407695390188 us; error 32 us; skew 500 ppm
I20260926 07:28:15.390547 11443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407695, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:15.394068 11443 init.cc:377] Logged in from keytab as oryx/127.2.147.193@KRBTEST.COM (short username oryx)
I20260926 07:28:15.395491 11443 webserver.cc:460] Webserver started at http://127.2.147.193:34173/ using document root <none> and password file <none>
I20260926 07:28:15.396147 11443 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:15.396284 11443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:15.401443 11443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:28:15.404062 11477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.405381 11443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:28:15.405542 11443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b8119c1032d3460ba6cd8e681a6de678"
format_stamp: "Formatted at 2026-09-26 07:28:12 on dist-test-slave-1dwd"
server_key: "83856a4a586d60fae7939c3789c1dc0c"
server_key_iv: "fc82f8eeb0b0a21447606f34c2c1d5e7"
server_key_version: "encryptionkey@0"
I20260926 07:28:15.406435 11443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:15.420406 11443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:15.424438 11443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:15.424707 11443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:15.425625 11443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:15.426920 11443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:15.427003 11443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.427145 11443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:15.427271 11443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.458189 11443 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45255
I20260926 07:28:15.458269 11589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45255 every 8 connection(s)
I20260926 07:28:15.459625 11443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:15.464918 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11443
I20260926 07:28:15.465659 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:40913
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:34941
--webserver_port=36809
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:28:15.781523 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:15.781829 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:15.781966 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:15.788978 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:15.789124 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:15.789284 11594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:15.798565 11594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.194.keytab
--principal=oryx/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:40913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=36809
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:15.799787 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:15.800992 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:15.817605 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:15.817754 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:15.818360 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:15.820206 11594 server_base.cc:1034] running on GCE node
I20260926 07:28:15.821156 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:15.822556 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:15.823833 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790407695823755 us; error 47 us; skew 500 ppm
I20260926 07:28:15.824334 11594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407695, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:15.829054 11594 init.cc:377] Logged in from keytab as oryx/127.2.147.194@KRBTEST.COM (short username oryx)
I20260926 07:28:15.830570 11594 webserver.cc:460] Webserver started at http://127.2.147.194:36809/ using document root <none> and password file <none>
I20260926 07:28:15.831247 11594 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:15.831391 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:15.837505 11594 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 07:28:15.840342 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.841631 11594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:28:15.841781 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "087371a85a04433d9730b03784de5454"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "d58af5f2e94a4a7f8520bd5f389838d0"
server_key_iv: "d9c76d6904ff023e0e85206de76e842a"
server_key_version: "encryptionkey@0"
I20260926 07:28:15.842662 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:15.860255 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:15.864707 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:15.864979 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:15.866080 11594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:15.867882 11594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:15.867975 11594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.868131 11594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:15.868198 11594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:15.910940 11594 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:40913
I20260926 07:28:15.911226 11722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:40913 every 8 connection(s)
I20260926 07:28:15.912474 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:15.919726 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11594
I20260926 07:28:15.920325 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:43059
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:34941
--webserver_port=44001
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:15 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:28:16.249560 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:16.249842 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:16.250007 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:16.257366 11727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:16.257505 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:16.257690 11727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:16.266911 11727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/krb5kdc/oryx_127.2.147.195.keytab
--principal=oryx/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:43059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=44001
--tserver_master_addrs=127.2.147.254:34941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:16.268190 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:16.269456 11727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:16.286669 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:16.292127 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:16.296687 11737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:16.297179 11727 server_base.cc:1034] running on GCE node
I20260926 07:28:16.297858 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:16.298893 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:16.300143 11727 hybrid_clock.cc:648] HybridClock initialized: now 1790407696300072 us; error 68 us; skew 500 ppm
I20260926 07:28:16.300496 11727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:16 dist-test-slave-1dwd krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407696, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:16.303643 11727 init.cc:377] Logged in from keytab as oryx/127.2.147.195@KRBTEST.COM (short username oryx)
I20260926 07:28:16.308473 11727 webserver.cc:460] Webserver started at http://127.2.147.195:44001/ using document root <none> and password file <none>
I20260926 07:28:16.309257 11727 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:16.309409 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:16.314966 11727 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:28:16.317673 11743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:16.319074 11727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:28:16.319274 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "78ea6554fc7b4b98a63e04a57c8aa235"
format_stamp: "Formatted at 2026-09-26 07:28:13 on dist-test-slave-1dwd"
server_key: "26f7f76200f72c69b178ef74f30b0ecc"
server_key_iv: "a2e77908c48c6d2db71daf1ecb943dd4"
server_key_version: "encryptionkey@0"
I20260926 07:28:16.320178 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:16.337392 11727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:16.341784 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:16.342043 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:16.343009 11727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:16.344568 11727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:16.344683 11727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:16.344810 11727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:16.344895 11727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:16.383842 11727 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:43059
I20260926 07:28:16.383888 11855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:43059 every 8 connection(s)
I20260926 07:28:16.385349 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:16.395394 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11727
Sep 26 07:28:16 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:16 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:16.425674 11859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:16.396733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.2.147.254:34941)
0926 07:28:16.403972 (+ 7239us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:16.403979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:16.404002 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:28:16.404093 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:16.404101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:16.404217 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:16.406860 (+ 2643us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:16.408320 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.409989 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.411556 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.412206 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.415617 (+ 3411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:16.415901 (+ 284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:16.415912 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:16.415930 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:16.415981 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:16.419976 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:16.421166 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:16.421173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:16.421180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:16.421362 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:16.421827 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:16.421834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:16.421838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:16.422423 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:16.423855 (+ 1432us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:16.424130 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:28:16.424568 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7112,"thread_start_us":6679,"threads_started":1}
W20260926 07:28:17.002405 11593 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.528s user 0.005s sys 0.011s
W20260926 07:28:17.004621 11592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:15.462382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:39303 (local address 127.2.147.254:34941)
0926 07:28:15.463166 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:15.463177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:15.463472 (+ 295us) server_negotiation.cc:407] Connection header received
0926 07:28:15.465923 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:15.465942 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:15.466258 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:15.466547 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:15.468516 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.469787 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.472236 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.473373 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.473572 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.004141 (+1530569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:39303: BlockingRecv error: recv got EOF from 127.2.147.193:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":357,"threads_started":1}
I20260926 07:28:17.005436 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:15.462463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:39303)
0926 07:28:15.463307 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:15.463356 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:15.465329 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:15.466754 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:15.466767 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:15.467573 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:15.468290 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:15.468315 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.469963 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.469971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:15.472083 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:15.472100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.473585 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.473590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:15.473767 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:15.474417 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:15.474442 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.003705 (+1529263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":339,"threads_started":1}
W20260926 07:28:17.007136 11590 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34941 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.147.254:34941: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:28:17.025626 11592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.008224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:53427 (local address 127.2.147.254:34941)
0926 07:28:17.008447 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.008457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.008582 (+ 125us) server_negotiation.cc:407] Connection header received
0926 07:28:17.008899 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.008932 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.009035 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.009128 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.011023 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.012611 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.015332 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.016327 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.016544 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.025240 (+ 8696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:53427: BlockingRecv error: recv got EOF from 127.2.147.193:53427 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:28:17.026607 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.008115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:53427)
0926 07:28:17.008465 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.008492 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.008718 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.009520 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.009527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.010439 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.010780 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.010796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.012881 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.012889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.015134 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.015152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.016876 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.016885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.017165 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.018157 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.018191 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.026213 (+ 8022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:28:17.048357 11592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.028067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:48951 (local address 127.2.147.254:34941)
0926 07:28:17.028293 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.028301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.033148 (+ 4847us) server_negotiation.cc:407] Connection header received
0926 07:28:17.033528 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.033536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.033662 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.033746 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.035441 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.037030 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.039648 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.040628 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.040800 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.047982 (+ 7182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:48951: BlockingRecv error: recv got EOF from 127.2.147.193:48951 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:28:17.049145 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.027627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:48951)
0926 07:28:17.033018 (+ 5391us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.033046 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.033347 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.033995 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.034002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.034802 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.035211 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.035269 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.037209 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.037217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.039462 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.039483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.040866 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.040876 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.041166 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.042064 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.042096 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.048665 (+ 6569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5187,"thread_start_us":4907,"threads_started":1}
W20260926 07:28:17.049429 11590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:28:17.366550 11726 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.435s user 0.006s sys 0.012s
W20260926 07:28:17.367820 11725 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:15.915459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:33881 (local address 127.2.147.254:34941)
0926 07:28:15.916011 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:15.916019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:15.917192 (+ 1173us) server_negotiation.cc:407] Connection header received
0926 07:28:15.921079 (+ 3887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:15.921087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:15.921179 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:15.921307 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:15.924176 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.925383 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.928846 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.929605 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.929750 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.367515 (+1437765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:33881: BlockingRecv error: recv got EOF from 127.2.147.194:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":277,"threads_started":1}
I20260926 07:28:17.368363 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:15.915897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:33881)
0926 07:28:15.916964 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:15.917042 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:15.920645 (+ 3603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:15.921690 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:15.921707 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:15.922778 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:15.923927 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:15.923962 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.925580 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.925589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:15.928652 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:15.928675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:15.929922 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:15.929932 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:15.930335 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:15.931363 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:15.931403 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.367335 (+1435932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":364,"threads_started":1}
W20260926 07:28:17.369156 11723 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34941 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.147.254:34941: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:17.379331 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.369821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:40191)
0926 07:28:17.370077 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.370096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.370205 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.370541 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.370546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.371032 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.371187 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.371195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.372367 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.372372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.373529 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.373540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.374237 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.374243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.374409 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.375163 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.375179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.379008 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 07:28:17.379431 11725 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.369906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:40191 (local address 127.2.147.254:34941)
0926 07:28:17.370071 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.370077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.370143 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:28:17.370334 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.370339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.370412 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.370518 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.371345 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.372234 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.373666 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.374061 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.374196 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.379177 (+ 4981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:40191: BlockingRecv error: recv got EOF from 127.2.147.194:40191 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:17.391865 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.380533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:58901)
0926 07:28:17.380837 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.380854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.381004 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.381481 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.381488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.381945 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.382136 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.382148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.383535 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.383543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.385026 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.385041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.386399 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.386405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.386589 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.387354 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.387377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.391697 (+ 4320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211}
W20260926 07:28:17.392163 11723 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:28:17.392452 11725 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.380674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58901 (local address 127.2.147.254:34941)
0926 07:28:17.380819 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.380825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.380927 (+ 102us) server_negotiation.cc:407] Connection header received
0926 07:28:17.381193 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.381199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.381282 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.381386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.382246 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.383306 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.385205 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.386220 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.386386 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.392018 (+ 5632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:58901: BlockingRecv error: recv got EOF from 127.2.147.194:58901 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 07:28:17.523386 11858 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.103s user 0.001s sys 0.014s
W20260926 07:28:17.525005 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:16.389921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:53645 (local address 127.2.147.254:34941)
0926 07:28:16.403653 (+ 13732us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:16.403662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:16.408291 (+ 4629us) server_negotiation.cc:407] Connection header received
0926 07:28:16.411687 (+ 3396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:16.411691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:16.411756 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:16.411823 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:16.414317 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.415696 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.418246 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.418920 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.419010 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.524559 (+1105549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:53645: BlockingRecv error: recv got EOF from 127.2.147.195:53645 (error 108)
Metrics: {"server-negotiator.queue_time_us":13588,"thread_start_us":13840,"threads_started":1}
I20260926 07:28:17.525530 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:16.389279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:53645)
0926 07:28:16.407813 (+ 18534us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:16.407865 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:16.411389 (+ 3524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:16.412428 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:16.412440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:16.413224 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:16.414100 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:16.414127 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.415894 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.415899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:16.418068 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:16.418085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:16.419375 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:16.419382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:16.419704 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:16.420415 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:16.420444 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.524340 (+1103896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18171,"spinlock_wait_cycles":19328,"thread_start_us":18040,"threads_started":1}
W20260926 07:28:17.526347 11856 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34941 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.147.254:34941: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:17.534802 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.526936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:42227)
0926 07:28:17.527151 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.527171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.527298 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.527660 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.527664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.528062 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.528217 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.528225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.529211 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.529216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.530165 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.530175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.530837 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.530843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.531011 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.531554 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.531570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.534628 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 07:28:17.534996 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.527000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:42227 (local address 127.2.147.254:34941)
0926 07:28:17.527150 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.527157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.527232 (+ 75us) server_negotiation.cc:407] Connection header received
0926 07:28:17.527430 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.527435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.527509 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.527622 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.528346 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.529070 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.530294 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.530679 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.530805 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.534761 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:42227: BlockingRecv error: recv got EOF from 127.2.147.195:42227 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:17 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:17.542876 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:17.535449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:33497)
0926 07:28:17.535612 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:17.535630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:17.535724 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:17.536105 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:17.536112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:17.536441 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:17.536576 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.536583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.537636 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.537639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.538549 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:17.538558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.539163 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.539167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:17.539346 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.539810 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:17.539825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:17.542622 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:28:17.543016 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:17.535631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:33497 (local address 127.2.147.254:34941)
0926 07:28:17.535861 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:17.535865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:17.535878 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:28:17.535911 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:17.535915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:17.535976 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:17.536044 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:17.536689 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.537518 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.538676 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:17.539049 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:17.539134 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:17.542789 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:33497: BlockingRecv error: recv got EOF from 127.2.147.195:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260926 07:28:17.543162 11856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:18.058167 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:18.050073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:57653)
0926 07:28:18.050538 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:18.050554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:18.050663 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:18.050949 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:18.050951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:18.051374 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:18.051533 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.051541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.052499 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.052503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.053397 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.053407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.053995 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.053999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.054111 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.054633 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:18.054649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:18.057955 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":255,"threads_started":1}
W20260926 07:28:18.058337 11864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:18.050077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:57653 (local address 127.2.147.254:34941)
0926 07:28:18.050515 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:18.050519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:18.050630 (+ 111us) server_negotiation.cc:407] Connection header received
0926 07:28:18.050755 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:18.050758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:18.050827 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:18.050913 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:18.051655 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.052350 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.053506 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.053859 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.053945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.058132 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:57653: BlockingRecv error: recv got EOF from 127.2.147.193:57653 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:18.401304 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:18.392663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:34465)
0926 07:28:18.393171 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:18.393186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:18.393289 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:18.393557 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:18.393560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:18.393946 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:18.394114 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.394140 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.395105 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.395110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.396203 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.396214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.397014 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.397022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.397190 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.397724 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:18.397739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:18.400985 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":298,"threads_started":1}
W20260926 07:28:18.401388 11864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:18.392705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:34465 (local address 127.2.147.254:34941)
0926 07:28:18.392869 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:18.392873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:18.393259 (+ 386us) server_negotiation.cc:407] Connection header received
0926 07:28:18.393382 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:18.393385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:18.393441 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:18.393519 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:18.394249 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.394988 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.396319 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.396827 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.396941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.401176 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:34465: BlockingRecv error: recv got EOF from 127.2.147.194:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:18 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:18.552732 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:18.543805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:39133)
0926 07:28:18.544244 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:18.544259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:18.544364 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:18.544643 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:18.544645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:18.545039 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:18.545172 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.545180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.546197 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.546201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.547408 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:18.547424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.548318 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.548324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:18.548512 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.549074 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:18.549090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:18.552394 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":256,"threads_started":1}
W20260926 07:28:18.552785 11864 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:18.543887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:39133 (local address 127.2.147.254:34941)
0926 07:28:18.544137 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:18.544143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:18.544336 (+ 193us) server_negotiation.cc:407] Connection header received
0926 07:28:18.544461 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:18.544464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:18.544530 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:18.544609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:18.545313 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.546075 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.547576 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:18.548139 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:18.548257 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:18.552566 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:39133: BlockingRecv error: recv got EOF from 127.2.147.195:39133 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:19.073411 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:19.059087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:59137)
0926 07:28:19.059842 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:19.059858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:19.059990 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:19.060367 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:19.060371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:19.060804 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:19.060946 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.060955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.061874 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.061877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.062843 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.062852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.063532 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.063539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.063677 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.069394 (+ 5717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:19.069412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:19.073115 (+ 3703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":474,"threads_started":1}
W20260926 07:28:19.073530 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:19.059145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:59137 (local address 127.2.147.254:34941)
0926 07:28:19.059937 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:19.059942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:19.059978 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:28:19.060126 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:19.060130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:19.060215 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:19.060334 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:19.061069 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.061751 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.062953 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.063418 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.063505 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.073301 (+ 9796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:59137: BlockingRecv error: recv got EOF from 127.2.147.193:59137 (error 108)
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":567,"threads_started":1}
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:19.410590 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:19.402281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:34099)
0926 07:28:19.402754 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:19.402768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:19.402871 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:19.403153 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:19.403155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:19.403565 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:19.403732 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.403741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.404638 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.404641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.405508 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.405517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.406146 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.406150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.406262 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.406810 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:19.406827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:19.410232 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":292,"threads_started":1}
W20260926 07:28:19.410691 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:19.402312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:34099 (local address 127.2.147.254:34941)
0926 07:28:19.402483 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:19.402487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:19.402840 (+ 353us) server_negotiation.cc:407] Connection header received
0926 07:28:19.402970 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:19.402974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:19.403027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:19.403102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:19.403850 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.404523 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.405621 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.406021 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.406106 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.410399 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:34099: BlockingRecv error: recv got EOF from 127.2.147.194:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:19 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:19.561920 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:19.553551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:43567)
0926 07:28:19.554059 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:19.554075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:19.554180 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:19.554507 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:19.554509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:19.554884 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:19.555058 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.555066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.556023 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.556026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.556962 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:19.556972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.557593 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.557597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:19.557713 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.558235 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:19.558250 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:19.561697 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":305,"threads_started":1}
W20260926 07:28:19.562111 11869 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:19.553710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43567 (local address 127.2.147.254:34941)
0926 07:28:19.553873 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:19.553877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:19.554147 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:28:19.554281 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:19.554284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:19.554354 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:19.554441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:19.555175 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.555904 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.557078 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:19.557464 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:19.557558 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:19.561858 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:43567: BlockingRecv error: recv got EOF from 127.2.147.195:43567 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:20.082990 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:20.074262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:36455)
0926 07:28:20.074750 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:20.074772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:20.074895 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:20.075272 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:20.075277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:20.075709 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:20.075867 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.075876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.076882 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.076887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.077769 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.077780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.078433 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.078438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.078603 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.079254 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:20.079271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:20.082642 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":205,"threads_started":1}
W20260926 07:28:20.083117 11872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:20.074286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:36455 (local address 127.2.147.254:34941)
0926 07:28:20.074853 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:20.074858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:20.074875 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:28:20.075033 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:20.075042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:20.075110 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:20.075239 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:20.076007 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.076728 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.077903 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.078297 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.078421 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.082799 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:36455: BlockingRecv error: recv got EOF from 127.2.147.193:36455 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":367,"threads_started":1}
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:20.420003 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:20.411641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:33255)
0926 07:28:20.412091 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:20.412106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:20.412260 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:20.412615 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:20.412621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:20.412989 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:20.413149 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.413158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.414157 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.414165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.415040 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.415050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.415722 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.415727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.415893 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.416405 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:20.416420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:20.419673 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
W20260926 07:28:20.420114 11872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:20.411686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:33255 (local address 127.2.147.254:34941)
0926 07:28:20.411912 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:20.411920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:20.412189 (+ 269us) server_negotiation.cc:407] Connection header received
0926 07:28:20.412390 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:20.412395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:20.412482 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:20.412579 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:20.413294 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.414022 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.415174 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.415580 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.415711 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.419872 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:33255: BlockingRecv error: recv got EOF from 127.2.147.194:33255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:20 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:20.570819 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:20.562799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:40589)
0926 07:28:20.563372 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:20.563388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:20.563495 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:20.563778 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:20.563781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:20.564142 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:20.564294 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.564302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.565195 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.565198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.566037 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:20.566046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.566679 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.566683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:20.566797 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.567400 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:20.567415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:20.570482 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
W20260926 07:28:20.571102 11872 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:20.562886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:40589 (local address 127.2.147.254:34941)
0926 07:28:20.563053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:20.563057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:20.563465 (+ 408us) server_negotiation.cc:407] Connection header received
0926 07:28:20.563594 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:20.563597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:20.563664 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:20.563740 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:20.564409 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.565088 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.566151 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:20.566567 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:20.566653 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:20.570740 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:40589: BlockingRecv error: recv got EOF from 127.2.147.195:40589 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:21.092494 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.083782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:33273)
0926 07:28:21.084263 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.084278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.084380 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.084698 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.084701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.085182 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.085404 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.085419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.086773 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.086776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.087863 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.087874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.088519 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.088523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.088632 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.089204 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.089220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.092236 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":282,"threads_started":1}
W20260926 07:28:21.092687 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:21.083816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:33273 (local address 127.2.147.254:34941)
0926 07:28:21.084262 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.084266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.084347 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:28:21.084469 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.084472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.084538 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.084649 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.085619 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.086654 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.087981 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.088372 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.088477 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.092436 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:33273: BlockingRecv error: recv got EOF from 127.2.147.193:33273 (error 108)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":270,"threads_started":1}
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:21.429174 11878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.420888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:59395)
0926 07:28:21.421330 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.421345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.421452 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.421726 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.421729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.422056 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.422232 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.422240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.423353 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.423357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.424417 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.424429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.425024 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.425028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.425150 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.425695 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.425711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.428864 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
W20260926 07:28:21.429330 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:21.420927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:59395 (local address 127.2.147.254:34941)
0926 07:28:21.421123 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.421133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.421422 (+ 289us) server_negotiation.cc:407] Connection header received
0926 07:28:21.421531 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.421535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.421595 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.421672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.422365 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.423161 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.424540 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.424910 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.425000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.429083 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:59395: BlockingRecv error: recv got EOF from 127.2.147.194:59395 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:21 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:21.580854 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.571738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:52447)
0926 07:28:21.572402 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.572418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.572603 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.572981 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.572985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.573363 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.573522 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.573530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.574526 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.574534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.575595 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.575606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.576307 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.576312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.576496 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.577049 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.577066 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.580519 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":388,"threads_started":1}
W20260926 07:28:21.581022 11877 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:21.571853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:52447 (local address 127.2.147.254:34941)
0926 07:28:21.572185 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.572192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.572525 (+ 333us) server_negotiation.cc:407] Connection header received
0926 07:28:21.572736 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.572741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.572822 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.572965 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.573662 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.574407 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.575726 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.576171 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.576297 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.580711 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:52447: BlockingRecv error: recv got EOF from 127.2.147.195:52447 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:22.101675 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:22.093330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:58527)
0926 07:28:22.093821 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:22.093844 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:22.093976 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:22.094401 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:22.094405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:22.094789 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:22.094921 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.094930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.095979 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.095986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.096914 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.096924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.097570 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.097575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.097742 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.098261 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:22.098277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:22.101344 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
W20260926 07:28:22.101752 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:22.093395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:58527 (local address 127.2.147.254:34941)
0926 07:28:22.094057 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:22.094064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:22.094080 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:28:22.094172 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:22.094177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:22.094251 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:22.094369 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:22.095054 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.095847 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.097050 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.097416 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.097535 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.101532 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:58527: BlockingRecv error: recv got EOF from 127.2.147.193:58527 (error 108)
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":446,"threads_started":1}
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:22.438694 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:22.430270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:45635)
0926 07:28:22.430725 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:22.430740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:22.430846 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:22.431153 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:22.431158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:22.431756 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:22.431973 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.431988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.432983 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.432986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.434118 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.434129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.434704 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.434707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.434816 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.435415 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:22.435433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:22.438487 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":248,"threads_started":1}
W20260926 07:28:22.438891 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:22.430483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:45635 (local address 127.2.147.254:34941)
0926 07:28:22.430645 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:22.430649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:22.430813 (+ 164us) server_negotiation.cc:407] Connection header received
0926 07:28:22.430952 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:22.430955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:22.431010 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:22.431097 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:22.432120 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.432865 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.434230 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.434595 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.434681 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.438655 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:45635: BlockingRecv error: recv got EOF from 127.2.147.194:45635 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:22 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:22.590545 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:22.581729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:49661)
0926 07:28:22.582211 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:22.582226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:22.582390 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:22.582730 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:22.582735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:22.583326 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:22.583525 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.583536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.584877 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.584881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.585795 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:22.585804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.586401 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.586405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:22.586519 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.587130 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:22.587147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:22.590331 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
W20260926 07:28:22.590775 11881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:22.581806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:49661 (local address 127.2.147.254:34941)
0926 07:28:22.582013 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:22.582021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:22.582306 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:28:22.582539 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:22.582542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:22.582609 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:22.582706 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:22.583679 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.584760 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.585915 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:22.586291 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:22.586377 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:22.590511 (+ 4134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:49661: BlockingRecv error: recv got EOF from 127.2.147.195:49661 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:23.112655 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:23.102710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:51505)
0926 07:28:23.103187 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:23.103228 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:23.103356 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:23.103807 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:23.103812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:23.104198 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:23.104338 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.104345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.105394 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.105399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.106682 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.106693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.107666 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.107671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.107839 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.108502 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:23.108550 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:23.112283 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":254,"threads_started":1}
W20260926 07:28:23.112780 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:23.102740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:51505 (local address 127.2.147.254:34941)
0926 07:28:23.103533 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:23.103537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:23.103549 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:28:23.103587 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:23.103590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:23.103651 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:23.103760 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:23.104470 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.105233 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.106805 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.107522 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.107641 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.112518 (+ 4877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:51505: BlockingRecv error: recv got EOF from 127.2.147.193:51505 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":562,"threads_started":1}
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:23.448629 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:23.439681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:44645)
0926 07:28:23.440161 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:23.440179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:23.440297 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:23.440584 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:23.440587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:23.440955 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:23.441100 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.441109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.442073 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.442076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.443039 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.443056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.444014 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.444019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.444211 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.444987 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:23.445003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:23.448391 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":276,"threads_started":1}
W20260926 07:28:23.448807 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:23.439748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:44645 (local address 127.2.147.254:34941)
0926 07:28:23.439930 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:23.439934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:23.440263 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:28:23.440391 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:23.440394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:23.440455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:23.440536 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:23.441216 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.441960 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.443180 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.443831 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.444005 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.448586 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:44645: BlockingRecv error: recv got EOF from 127.2.147.194:44645 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:23 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:23.600472 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:23.591578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:52521)
0926 07:28:23.592038 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:23.592058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:23.592173 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:23.592541 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:23.592544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:23.592929 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:23.593137 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.593150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.594620 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.594624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.595601 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:23.595613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.596234 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.596241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:23.596405 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.596944 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:23.596959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:23.600139 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":257,"threads_started":1}
W20260926 07:28:23.600550 11885 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:23.591715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:52521 (local address 127.2.147.254:34941)
0926 07:28:23.591901 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:23.591905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:23.592139 (+ 234us) server_negotiation.cc:407] Connection header received
0926 07:28:23.592304 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:23.592307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:23.592404 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:23.592495 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:23.593283 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.594481 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.595712 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:23.596076 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:23.596203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:23.600307 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:52521: BlockingRecv error: recv got EOF from 127.2.147.195:52521 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:24.125901 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:24.113738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:42871)
0926 07:28:24.114285 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:24.114303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:24.114471 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:24.114950 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:24.114957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:24.115559 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:24.115844 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.115855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.117162 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.117167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.118815 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.118831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.119963 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.119969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.120341 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.120952 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:24.120977 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:24.125537 (+ 4560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":242,"threads_started":1}
W20260926 07:28:24.125972 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:24.113816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:42871 (local address 127.2.147.254:34941)
0926 07:28:24.114268 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:24.114274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:24.114373 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:28:24.114609 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:24.114618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:24.114699 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:24.114813 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:24.116000 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.116982 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.119059 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.119807 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.119930 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.125719 (+ 5789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:42871: BlockingRecv error: recv got EOF from 127.2.147.193:42871 (error 108)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:24.458022 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:24.449550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:52187)
0926 07:28:24.450047 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:24.450062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:24.450171 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:24.450447 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:24.450450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:24.450834 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:24.451022 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.451031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.452032 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.452036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.452995 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.453006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.453602 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.453606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.453717 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.454353 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:24.454370 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:24.457679 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":219,"threads_started":1}
W20260926 07:28:24.458096 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:24.449642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:52187 (local address 127.2.147.254:34941)
0926 07:28:24.449900 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:24.449906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:24.450143 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:28:24.450266 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:24.450270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:24.450327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:24.450403 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:24.451136 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.451923 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.453112 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.453477 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.453570 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.457870 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:52187: BlockingRecv error: recv got EOF from 127.2.147.194:52187 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:24 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:24.610543 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:24.601642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:59523)
0926 07:28:24.602260 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:24.602276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:24.602458 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:24.602814 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:24.602818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:24.603286 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:24.603449 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.603458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.604459 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.604464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.605508 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:24.605519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.606132 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.606136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:24.606245 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.606864 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:24.606880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:24.610236 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":309,"threads_started":1}
W20260926 07:28:24.611011 11889 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:24.601720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:59523 (local address 127.2.147.254:34941)
0926 07:28:24.601931 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:24.601938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:24.602382 (+ 444us) server_negotiation.cc:407] Connection header received
0926 07:28:24.602600 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:24.602605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:24.602681 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:24.602781 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:24.603598 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.604280 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.605621 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:24.606002 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:24.606101 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:24.610520 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:59523: BlockingRecv error: recv got EOF from 127.2.147.195:59523 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:25.135253 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:25.126700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:39905)
0926 07:28:25.127269 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:25.127287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:25.127405 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:25.127697 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:25.127701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:25.128088 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:25.128242 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.128250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.129165 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.129168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.130080 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.130090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.130758 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.130762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.130865 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.131446 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:25.131462 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:25.134896 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
W20260926 07:28:25.135469 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:25.126859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:39905 (local address 127.2.147.254:34941)
0926 07:28:25.127296 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:25.127300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:25.127361 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:28:25.127495 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:25.127499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:25.127576 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:25.127664 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:25.128354 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.129053 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.130191 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.130609 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.130709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.135088 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:39905: BlockingRecv error: recv got EOF from 127.2.147.193:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":221,"threads_started":1}
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:25.468037 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:25.459067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:60191)
0926 07:28:25.459584 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:25.459600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:25.459755 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:25.460072 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:25.460076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:25.460563 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:25.460799 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.460813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.462241 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.462245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.463170 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.463180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.463853 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.463857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.463963 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.464575 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:25.464590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:25.467684 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
W20260926 07:28:25.468065 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:25.459142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:60191 (local address 127.2.147.254:34941)
0926 07:28:25.459331 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:25.459337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:25.459683 (+ 346us) server_negotiation.cc:407] Connection header received
0926 07:28:25.459871 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:25.459876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:25.459940 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:25.460047 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:25.460950 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.462122 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.463328 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.463718 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.463805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.467850 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:60191: BlockingRecv error: recv got EOF from 127.2.147.194:60191 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:25 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:25.620249 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:25.611454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:52009)
0926 07:28:25.611903 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:25.611919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:25.612068 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:25.612460 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:25.612464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:25.612860 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:25.613014 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.613022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.614085 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.614091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.615018 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:25.615029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.615800 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.615806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:25.615962 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.616500 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:25.616516 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:25.619906 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
W20260926 07:28:25.620326 11893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:25.611504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:52009 (local address 127.2.147.254:34941)
0926 07:28:25.611692 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:25.611699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:25.612005 (+ 306us) server_negotiation.cc:407] Connection header received
0926 07:28:25.612192 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:25.612197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:25.612272 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:25.612404 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:25.613171 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.613950 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.615155 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:25.615631 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:25.615765 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:25.620085 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:52009: BlockingRecv error: recv got EOF from 127.2.147.195:52009 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:26.144837 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:26.136129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:40251)
0926 07:28:26.137127 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:26.137152 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:26.137302 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:26.137585 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:26.137588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:26.137937 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:26.138072 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.138080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.139057 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.139061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.140074 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.140085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.140746 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.140750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.140863 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.141415 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:26.141432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:26.144612 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":460,"threads_started":1}
W20260926 07:28:26.144975 11896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:26.136159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:40251 (local address 127.2.147.254:34941)
0926 07:28:26.137373 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:26.137377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:26.137391 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:28:26.137430 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:26.137433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:26.137483 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:26.137573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:26.138208 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.138945 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.140186 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.140610 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.140715 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.144778 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:40251: BlockingRecv error: recv got EOF from 127.2.147.193:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":1055,"threads_started":1}
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:26.477766 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:26.469065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:60513)
0926 07:28:26.469548 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:26.469562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:26.469740 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:26.470097 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:26.470103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:26.470516 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:26.470654 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.470663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.471735 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.471742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.472757 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.472768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.473515 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.473521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.473700 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.474222 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:26.474237 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:26.477458 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":231,"threads_started":1}
W20260926 07:28:26.477809 11896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:26.469166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:60513 (local address 127.2.147.254:34941)
0926 07:28:26.469373 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:26.469380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:26.469666 (+ 286us) server_negotiation.cc:407] Connection header received
0926 07:28:26.469863 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:26.469868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:26.469965 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:26.470064 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:26.470803 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.471601 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.472901 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.473366 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.473488 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.477606 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:60513: BlockingRecv error: recv got EOF from 127.2.147.194:60513 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:26 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:26.630159 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:26.621491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:38469)
0926 07:28:26.622012 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:26.622029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:26.622144 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:26.622446 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:26.622449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:26.622806 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:26.622940 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.622948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.624022 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.624027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.625095 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:26.625106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.625739 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.625743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:26.625854 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.626404 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:26.626434 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:26.629825 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":287,"threads_started":1}
W20260926 07:28:26.630255 11896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:26.621548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:38469 (local address 127.2.147.254:34941)
0926 07:28:26.621715 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:26.621720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:26.622101 (+ 381us) server_negotiation.cc:407] Connection header received
0926 07:28:26.622256 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:26.622260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:26.622325 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:26.622413 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:26.623063 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.623899 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.625214 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:26.625616 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:26.625701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:26.629993 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:38469: BlockingRecv error: recv got EOF from 127.2.147.195:38469 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:27.154016 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:27.145726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:39771)
0926 07:28:27.146199 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:27.146215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:27.146312 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:27.146667 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:27.146670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:27.146988 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:27.147129 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.147136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.148059 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.148062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.148962 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.148972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.149597 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.149601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.149710 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.150266 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:27.150284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:27.153727 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
W20260926 07:28:27.154256 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:27.145808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:39771 (local address 127.2.147.254:34941)
0926 07:28:27.146236 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:27.146240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:27.146260 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:28:27.146418 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:27.146422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:27.146545 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:27.146637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:27.147278 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.147936 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.149071 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.149444 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.149546 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.153966 (+ 4420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:39771: BlockingRecv error: recv got EOF from 127.2.147.193:39771 (error 108)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":258,"threads_started":1}
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:27.487409 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:27.478612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:54257)
0926 07:28:27.479023 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:27.479039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:27.479144 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:27.479464 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:27.479468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:27.479806 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:27.479943 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.479950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.481238 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.481242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.482499 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.482511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.483153 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.483157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.483327 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.483853 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:27.483869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:27.487066 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":214,"threads_started":1}
W20260926 07:28:27.487473 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:27.478690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:54257 (local address 127.2.147.254:34941)
0926 07:28:27.478923 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:27.478929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:27.479112 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:28:27.479293 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:27.479297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:27.479353 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:27.479428 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:27.480061 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.481093 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.482635 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.483038 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.483123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.487262 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:54257: BlockingRecv error: recv got EOF from 127.2.147.194:54257 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:27 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:27.639627 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:27.631183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:37023)
0926 07:28:27.631676 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:27.631691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:27.631796 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:27.632073 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:27.632076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:27.632437 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:27.632592 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.632600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.633530 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.633533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.634433 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:27.634442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.635156 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.635162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:27.635364 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.636018 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:27.636037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:27.639284 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":237,"threads_started":1}
W20260926 07:28:27.639709 11901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:27.631333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:37023 (local address 127.2.147.254:34941)
0926 07:28:27.631578 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:27.631584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:27.631762 (+ 178us) server_negotiation.cc:407] Connection header received
0926 07:28:27.631890 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:27.631893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:27.631960 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:27.632045 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:27.632718 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.633409 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.634569 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:27.635004 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:27.635141 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:27.639447 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:37023: BlockingRecv error: recv got EOF from 127.2.147.195:37023 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:28.163671 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:28.155123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:43789)
0926 07:28:28.155643 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:28.155658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:28.155767 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:28.156104 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:28.156107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:28.156485 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:28.156656 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.156663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.157572 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.157575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.158429 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.158438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.159080 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.159084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.159187 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.159906 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:28.159930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:28.163453 (+ 3523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":304,"threads_started":1}
W20260926 07:28:28.163906 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:28.155231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43789 (local address 127.2.147.254:34941)
0926 07:28:28.155654 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:28.155658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:28.155732 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:28:28.155898 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:28.155902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:28.155979 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:28.156065 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:28.156773 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.157471 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.158547 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.158955 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.159042 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.163627 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:43789: BlockingRecv error: recv got EOF from 127.2.147.193:43789 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":246,"threads_started":1}
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:28.496271 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:28.488246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:34397)
0926 07:28:28.488720 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:28.488736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:28.488842 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:28.489116 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:28.489119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:28.489501 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:28.489676 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.489684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.490631 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.490635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.491539 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.491550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.492169 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.492173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.492279 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.492838 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:28.492854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:28.496061 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":262,"threads_started":1}
W20260926 07:28:28.496363 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:28.488352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:34397 (local address 127.2.147.254:34941)
0926 07:28:28.488508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:28.488512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:28.488807 (+ 295us) server_negotiation.cc:407] Connection header received
0926 07:28:28.488937 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:28.488941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:28.488998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:28.489076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:28.489800 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.490526 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.491657 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.492042 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.492139 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.496200 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:34397: BlockingRecv error: recv got EOF from 127.2.147.194:34397 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:28 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:28.648562 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:28.640489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:51793)
0926 07:28:28.640889 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:28.640905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:28.641012 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:28.641287 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:28.641290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:28.641675 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:28.641848 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.641855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.642753 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.642756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.643655 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:28.643665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.644271 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.644275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:28.644385 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.644948 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:28.644964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:28.648289 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":228,"threads_started":1}
W20260926 07:28:28.648739 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:28.640528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:51793 (local address 127.2.147.254:34941)
0926 07:28:28.640748 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:28.640754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:28.640980 (+ 226us) server_negotiation.cc:407] Connection header received
0926 07:28:28.641114 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:28.641117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:28.641174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:28.641251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:28.641959 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.642653 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.643764 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:28.644152 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:28.644240 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:28.648489 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:51793: BlockingRecv error: recv got EOF from 127.2.147.195:51793 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:29.178911 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:29.164584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:38355)
0926 07:28:29.165112 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:29.165128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:29.165220 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:29.165672 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:29.165674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:29.166065 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:29.166217 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.166224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.167155 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.167159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.168090 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.168102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.168693 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.168697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.168817 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.175577 (+ 6760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:29.175594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:29.178676 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":331,"threads_started":1}
W20260926 07:28:29.179080 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:29.164619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:38355 (local address 127.2.147.254:34941)
0926 07:28:29.165447 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:29.165451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:29.165465 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:28:29.165503 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:29.165506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:29.165554 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:29.165632 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:29.166333 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.167024 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.168209 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.168575 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.168655 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.178851 (+ 10196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:38355: BlockingRecv error: recv got EOF from 127.2.147.193:38355 (error 108)
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":650,"threads_started":1}
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:29.505784 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:29.497134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:41487)
0926 07:28:29.497618 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:29.497634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:29.497743 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:29.498023 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:29.498026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:29.498483 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:29.498649 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.498658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.499660 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.499663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.500565 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.500575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.501499 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.501504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.501665 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.502363 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:29.502387 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:29.505551 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":272,"threads_started":1}
W20260926 07:28:29.505872 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:29.497218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:41487 (local address 127.2.147.254:34941)
0926 07:28:29.497403 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:29.497408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:29.497706 (+ 298us) server_negotiation.cc:407] Connection header received
0926 07:28:29.497837 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:29.497841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:29.497895 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:29.497974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:29.498787 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.499547 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.500708 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.501292 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.501450 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.505710 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:41487: BlockingRecv error: recv got EOF from 127.2.147.194:41487 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:29 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:29.657812 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:29.649426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:54509)
0926 07:28:29.649971 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:29.649986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:29.650139 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:29.650471 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:29.650475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:29.650889 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:29.651047 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.651056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.652063 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.652070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.653001 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:29.653011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.653771 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.653777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:29.653952 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.654565 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:29.654580 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:29.657585 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":289,"threads_started":1}
W20260926 07:28:29.657994 11909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:29.649496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:54509 (local address 127.2.147.254:34941)
0926 07:28:29.649709 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:29.649718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:29.650071 (+ 353us) server_negotiation.cc:407] Connection header received
0926 07:28:29.650272 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:29.650277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:29.650331 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:29.650433 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:29.651197 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.651930 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.653143 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:29.653604 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:29.653742 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:29.657752 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:54509: BlockingRecv error: recv got EOF from 127.2.147.195:54509 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:30.188675 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:30.179837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:43605)
0926 07:28:30.180276 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:30.180291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:30.180390 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:30.180902 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:30.180907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:30.181400 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:30.181570 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.181583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.183007 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.183011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.184037 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.184047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.184656 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.184660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.184780 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.185335 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:30.185351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:30.188457 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":244,"threads_started":1}
W20260926 07:28:30.188880 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:30.179923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43605 (local address 127.2.147.254:34941)
0926 07:28:30.180571 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:30.180575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:30.180590 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:28:30.180629 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:30.180632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:30.180698 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:30.180807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:30.181743 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.182887 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.184153 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.184548 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.184633 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.188632 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:43605: BlockingRecv error: recv got EOF from 127.2.147.193:43605 (error 108)
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":454,"threads_started":1}
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.194@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:30.515094 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:30.506690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.194:40685)
0926 07:28:30.507166 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:30.507182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:30.507336 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:30.507704 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:30.507708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:30.508139 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:30.508281 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.508290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.509301 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.509304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.510326 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.510337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.511021 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.511027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.511152 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.511769 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:30.511785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:30.514872 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
W20260926 07:28:30.515259 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:30.506861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:40685 (local address 127.2.147.254:34941)
0926 07:28:30.507041 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:30.507048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:30.507256 (+ 208us) server_negotiation.cc:407] Connection header received
0926 07:28:30.507462 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:30.507467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:30.507554 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:30.507649 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:30.508405 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.509183 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.510448 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.510871 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.510958 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.515033 (+ 4075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.194:40685: BlockingRecv error: recv got EOF from 127.2.147.194:40685 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:30 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.195@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:30.667181 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:30.658643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.195:44501)
0926 07:28:30.659111 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:30.659136 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:30.659298 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:30.659631 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:30.659636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:30.660079 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:30.660221 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.660229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.661251 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.661256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.662166 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:30.662176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.662842 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.662847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:30.663022 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.663683 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:30.663700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:30.666857 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":252,"threads_started":1}
W20260926 07:28:30.667387 11913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:30.658680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:44501 (local address 127.2.147.254:34941)
0926 07:28:30.658883 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:30.658890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:30.659235 (+ 345us) server_negotiation.cc:407] Connection header received
0926 07:28:30.659435 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:30.659440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:30.659507 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:30.659601 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:30.660371 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.661116 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.662317 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:30.662700 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:30.662820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:30.667057 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.195:44501: BlockingRecv error: recv got EOF from 127.2.147.195:44501 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 07:28:31 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:28:31 dist-test-slave-1dwd krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.147.193@KRBTEST.COM for oryx/127.2.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:28:31.198518 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:31.189725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34941 (local address 127.2.147.193:55807)
0926 07:28:31.190203 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:31.190219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:31.190314 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:31.190712 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:31.190715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:31.191055 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:31.191275 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:31.191287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.192406 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.192411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:31.193343 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:31.193354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.194003 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.194007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:31.194132 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:31.194753 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:31.194769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:31.198145 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.147.254:34941: Server oryx/127.2.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
W20260926 07:28:31.198601 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:31.189814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:55807 (local address 127.2.147.254:34941)
0926 07:28:31.190468 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:31.190473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:31.190487 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:28:31.190527 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:31.190530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:31.190592 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:31.190676 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:31.191481 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.192225 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.193461 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.193865 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.193950 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:31.198336 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.147.193:55807: BlockingRecv error: recv got EOF from 127.2.147.193:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":396,"threads_started":1}
I20260926 07:28:31.399545 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11443
I20260926 07:28:31.412357 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:31.400805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.2.147.193:45255)
0926 07:28:31.401476 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:31.401483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:31.401509 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:28:31.401571 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:31.401577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:31.401815 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:31.401994 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:31.403652 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.404699 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.405945 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.406495 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.406595 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:31.408982 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:31.408997 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:31.409009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:31.409053 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:31.411789 (+ 2736us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:28:31.412065 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42254: Request ticket server kudu/127.2.147.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":367,"threads_started":1}
W20260926 07:28:31.412463 11919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42254: Request ticket server kudu/127.2.147.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 07:28:31.412444 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11443: Not authorized: Client connection negotiation failed: client connection to 127.2.147.193:45255: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:28:31.412659 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11443
I20260926 07:28:31.429571 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11594
I20260926 07:28:31.441797 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:31.430682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.2.147.194:40913)
0926 07:28:31.431320 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:31.431327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:31.431353 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:28:31.431418 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:31.431424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:31.431677 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:31.431880 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:31.433171 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.434171 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.435382 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.435916 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.436063 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:31.438628 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:31.438636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:31.438649 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:31.438708 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:31.441101 (+ 2393us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:28:31.441466 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40144: Request ticket server kudu/127.2.147.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
W20260926 07:28:31.441785 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11594: Not authorized: Client connection negotiation failed: client connection to 127.2.147.194:40913: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:28:31.441927 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11594
W20260926 07:28:31.441907 11920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40144: Request ticket server kudu/127.2.147.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:28:31.457674 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11727
I20260926 07:28:31.469170 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:31.458697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.2.147.195:43059)
0926 07:28:31.459195 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:31.459245 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:31.459281 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:28:31.459343 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:31.459349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:31.459595 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:31.459795 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:31.461091 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.462045 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.463256 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:31.463811 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:31.463952 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:31.466230 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:31.466237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:31.466250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:31.466296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:31.468538 (+ 2242us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 07:28:31.468853 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57562: Request ticket server kudu/127.2.147.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
W20260926 07:28:31.469161 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11727: Not authorized: Client connection negotiation failed: client connection to 127.2.147.195:43059: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.147.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 07:28:31.469280 11921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57562: Request ticket server kudu/127.2.147.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 07:28:31.469314 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11727
I20260926 07:28:31.485637 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11374
I20260926 07:28:31.486950 11436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:31.600654 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11374
2026-09-26T07:28:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20869 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:31Z Disabled control of system clock
I20260926 07:28:31.673558 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39649
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:39649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:28:31.977214 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:31.977468 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:31.977603 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:31.984280 11926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:31.984400 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:31.984460 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:31.984561 11926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:31.984634 11926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:31.993531 11926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:39649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:31.994726 11926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:31.996486 11926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:32.008117 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.008061 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.008332 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:32.008431 11926 server_base.cc:1034] running on GCE node
I20260926 07:28:32.009409 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:32.010689 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:32.011874 11926 hybrid_clock.cc:648] HybridClock initialized: now 1790407712011835 us; error 32 us; skew 500 ppm
I20260926 07:28:32.012153 11926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:32.014137 11926 webserver.cc:460] Webserver started at http://127.2.147.254:42765/ using document root <none> and password file <none>
I20260926 07:28:32.014734 11926 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:32.014808 11926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:32.015036 11926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:32.017763 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "4b229448e1b644348c3fc7fba3bdeefa"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "f29e3bc2ccc3ca9668e405a37ad1ddf1"
server_key_iv: "430aed76e4a4d1b2cca2b441222c273f"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.018443 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "4b229448e1b644348c3fc7fba3bdeefa"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "f29e3bc2ccc3ca9668e405a37ad1ddf1"
server_key_iv: "430aed76e4a4d1b2cca2b441222c273f"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.023104 11926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:28:32.026189 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:32.027566 11926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:28:32.027729 11926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "4b229448e1b644348c3fc7fba3bdeefa"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "f29e3bc2ccc3ca9668e405a37ad1ddf1"
server_key_iv: "430aed76e4a4d1b2cca2b441222c273f"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.027936 11926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:32.058224 11926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:32.059114 11926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:32.059427 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:32.076741 11926 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:39649
I20260926 07:28:32.076761 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:39649 every 8 connection(s)
I20260926 07:28:32.077993 11926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:32.081274 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 11926
I20260926 07:28:32.081462 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:28:32.081712 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:32.082266 2639 external_mini_cluster.cc:1428] Setting key d8b411e8e6e9e0bc42ce2f8950fbf7db
I20260926 07:28:32.094520 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa: Bootstrap starting.
I20260926 07:28:32.096181 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:32.084366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.2.147.254:39649)
0926 07:28:32.085390 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:32.085406 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:32.085449 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:28:32.087616 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:32.087637 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:32.088073 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:32.088386 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:32.088949 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.090160 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.092040 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.092691 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.092820 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:32.094029 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:32.094036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:32.094048 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:32.094540 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:32.094649 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:32.094681 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:32.094798 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:32.095004 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 07:28:32.095320 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":409,"threads_started":1}
I20260926 07:28:32.097554 11992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:32.098547 11992 log.cc:824] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:32.101222 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa: No bootstrap required, opened a new log
I20260926 07:28:32.108434 11992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } }
I20260926 07:28:32.108721 11992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:32.108785 11992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b229448e1b644348c3fc7fba3bdeefa, State: Initialized, Role: FOLLOWER
I20260926 07:28:32.109210 11992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } }
I20260926 07:28:32.109417 11992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:32.109508 11992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:32.109623 11992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:32.111644 11992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } }
I20260926 07:28:32.112046 11992 leader_election.cc:302] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b229448e1b644348c3fc7fba3bdeefa; no voters:
I20260926 07:28:32.112608 11992 leader_election.cc:288] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:32.112754 11997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:32.113415 11997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [term 1 LEADER]: Becoming Leader. State: Replica: 4b229448e1b644348c3fc7fba3bdeefa, State: Running, Role: LEADER
I20260926 07:28:32.113938 11997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } }
I20260926 07:28:32.114584 11992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:32.116528 11998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b229448e1b644348c3fc7fba3bdeefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } } }
I20260926 07:28:32.116895 11998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:32.116647 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b229448e1b644348c3fc7fba3bdeefa. Latest consensus state: current_term: 1 leader_uuid: "4b229448e1b644348c3fc7fba3bdeefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b229448e1b644348c3fc7fba3bdeefa" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 39649 } } }
I20260926 07:28:32.117059 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:32.119302 12003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:32.124228 12003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:32.132839 12003 catalog_manager.cc:1347] Generated new cluster ID: c07a5638a266467c86fc5a8fe5ea3009
I20260926 07:28:32.132967 12003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:32.148932 12003 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:32.150125 12003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:32.160710 12003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b229448e1b644348c3fc7fba3bdeefa: Generated new TSK 0
I20260926 07:28:32.161474 12003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:28:32.173098 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--builtin_ntp_servers=127.2.147.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:28:32.521658 12016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:32.521907 12016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:32.521996 12016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:32.529076 12016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:32.529166 12016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:32.529268 12016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:32.538745 12016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:32.540400 12016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:32.542091 12016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:32.554684 12021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.554765 12024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.554736 12022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:32.555461 12016 server_base.cc:1034] running on GCE node
I20260926 07:28:32.556071 12016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:32.556933 12016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:32.558128 12016 hybrid_clock.cc:648] HybridClock initialized: now 1790407712558088 us; error 33 us; skew 500 ppm
I20260926 07:28:32.558429 12016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:32.560580 12016 webserver.cc:460] Webserver started at http://127.2.147.193:44817/ using document root <none> and password file <none>
I20260926 07:28:32.561245 12016 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:32.561355 12016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:32.561674 12016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:32.564381 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "80c352abe339458d88c478b4842d4796"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "a43e45672726924d95e3361858b81e1b"
server_key_iv: "cf9337d3c4c291d6412d737b91f18228"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.565130 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "80c352abe339458d88c478b4842d4796"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "a43e45672726924d95e3361858b81e1b"
server_key_iv: "cf9337d3c4c291d6412d737b91f18228"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.569880 12016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:28:32.573082 12030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:32.574295 12016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:28:32.574450 12016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "80c352abe339458d88c478b4842d4796"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "a43e45672726924d95e3361858b81e1b"
server_key_iv: "cf9337d3c4c291d6412d737b91f18228"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.574646 12016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:32.590409 12016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:32.591140 12016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:32.591466 12016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:32.592309 12016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:32.593626 12016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:32.593732 12016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:32.593856 12016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:32.593956 12016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:32.625885 12016 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:34421
I20260926 07:28:32.625931 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:34421 every 8 connection(s)
I20260926 07:28:32.627156 12016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:32.637102 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12016
I20260926 07:28:32.637310 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:32.637949 2639 external_mini_cluster.cc:1428] Setting key 8e146f4d0d0cb867bfc91c3272923431
I20260926 07:28:32.638746 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:32.629530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60417 (local address 127.2.147.254:39649)
0926 07:28:32.629940 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:32.629944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:32.631002 (+ 1058us) server_negotiation.cc:407] Connection header received
0926 07:28:32.632749 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:32.632754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:32.632816 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:32.632909 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:32.633847 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.634526 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.636011 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.636371 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.636454 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:32.637917 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:32.637922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:32.637925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:32.637943 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:32.637977 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:32.637988 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:32.638081 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:32.638295 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:28:32.638384 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":227,"threads_started":1}
I20260926 07:28:32.639829 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:32.630010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39649 (local address 127.2.147.193:60417)
0926 07:28:32.630829 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:32.630869 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:32.632523 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:32.633103 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:32.633111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:32.633190 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:32.633718 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:32.633733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.634653 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.634656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:32.635892 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:32.635903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:32.636530 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:32.636534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:32.636771 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:32.637613 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:32.637615 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:32.637651 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:32.637719 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:32.638081 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:32.638093 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:32.638105 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:28:32.638286 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 07:28:32.638650 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
I20260926 07:28:32.640622 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--builtin_ntp_servers=127.2.147.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:28:32.641232 12143 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39649
I20260926 07:28:32.641530 12143 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:32.642206 12143 heartbeater.cc:511] Master 127.2.147.254:39649 requested a full tablet report, sending...
I20260926 07:28:32.644305 11957 ts_manager.cc:194] Registered new tserver with Master: 80c352abe339458d88c478b4842d4796 (127.2.147.193:34421)
I20260926 07:28:32.646411 11957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.193:60417
W20260926 07:28:32.953212 12147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:32.953467 12147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:32.953544 12147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:32.960350 12147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:32.960435 12147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:32.960538 12147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:32.968940 12147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:32.970038 12147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:32.971676 12147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:32.983695 12155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.983740 12152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:32.983763 12153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:32.984755 12147 server_base.cc:1034] running on GCE node
I20260926 07:28:32.985270 12147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:32.986152 12147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:32.987370 12147 hybrid_clock.cc:648] HybridClock initialized: now 1790407712987340 us; error 44 us; skew 500 ppm
I20260926 07:28:32.987766 12147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:32.989893 12147 webserver.cc:460] Webserver started at http://127.2.147.194:37979/ using document root <none> and password file <none>
I20260926 07:28:32.990577 12147 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:32.990684 12147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:32.990998 12147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:32.993700 12147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "c497fe4d4af44125bf2a058667cc8e0a"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "c320efbf9427f0beab266fb268c86139"
server_key_iv: "b3c6d82e9e95aa3a0b3d598c56f149ed"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.994419 12147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "c497fe4d4af44125bf2a058667cc8e0a"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "c320efbf9427f0beab266fb268c86139"
server_key_iv: "b3c6d82e9e95aa3a0b3d598c56f149ed"
server_key_version: "encryptionkey@0"
I20260926 07:28:32.998925 12147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:28:33.002117 12161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.003381 12147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:33.003536 12147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "c497fe4d4af44125bf2a058667cc8e0a"
format_stamp: "Formatted at 2026-09-26 07:28:32 on dist-test-slave-1dwd"
server_key: "c320efbf9427f0beab266fb268c86139"
server_key_iv: "b3c6d82e9e95aa3a0b3d598c56f149ed"
server_key_version: "encryptionkey@0"
I20260926 07:28:33.003718 12147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:33.015075 12147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:33.015856 12147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:33.016182 12147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:33.017050 12147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:33.018314 12147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:33.018433 12147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.018551 12147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:33.018643 12147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.050241 12147 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:39699
I20260926 07:28:33.050325 12273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:39699 every 8 connection(s)
I20260926 07:28:33.051522 12147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:33.052570 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12147
I20260926 07:28:33.052722 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:33.053259 2639 external_mini_cluster.cc:1428] Setting key e90ac595be0dda94810c459842e24b13
I20260926 07:28:33.056180 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--builtin_ntp_servers=127.2.147.212:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:28:33.066088 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.054167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:43783 (local address 127.2.147.254:39649)
0926 07:28:33.054413 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.054420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.056065 (+ 1645us) server_negotiation.cc:407] Connection header received
0926 07:28:33.058773 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.058778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.058839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.058920 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.060272 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.060973 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.063360 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.063951 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.064084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.065294 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:33.065300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:33.065316 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:33.065331 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:33.065360 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:33.065372 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:33.065463 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:33.065682 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:28:33.065768 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:28:33.067114 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.054773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39649 (local address 127.2.147.194:43783)
0926 07:28:33.055885 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:33.055939 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:33.058507 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:33.059165 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:33.059177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:33.059318 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:33.060073 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:33.060098 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.061116 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.061124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:33.063221 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:33.063237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.064148 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.064153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:33.064326 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.064964 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:33.064966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:33.065013 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:33.065080 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:33.065461 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:33.065472 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:33.065482 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:28:33.065664 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 07:28:33.066041 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":363,"threads_started":1}
I20260926 07:28:33.068431 12274 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39649
I20260926 07:28:33.068717 12274 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:33.069406 12274 heartbeater.cc:511] Master 127.2.147.254:39649 requested a full tablet report, sending...
I20260926 07:28:33.070760 11957 ts_manager.cc:194] Registered new tserver with Master: c497fe4d4af44125bf2a058667cc8e0a (127.2.147.194:39699)
I20260926 07:28:33.071985 11957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.194:43783
W20260926 07:28:33.373453 12277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:33.373703 12277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:33.373811 12277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:33.380879 12277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:33.380967 12277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:33.381068 12277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:33.390292 12277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:33.391518 12277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:33.393172 12277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:33.406090 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:33.406029 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:33.406558 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:33.408651 12277 server_base.cc:1034] running on GCE node
I20260926 07:28:33.409247 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:33.410089 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:33.411295 12277 hybrid_clock.cc:648] HybridClock initialized: now 1790407713411239 us; error 46 us; skew 500 ppm
I20260926 07:28:33.411602 12277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:33.413684 12277 webserver.cc:460] Webserver started at http://127.2.147.195:41499/ using document root <none> and password file <none>
I20260926 07:28:33.414345 12277 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:33.414453 12277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:33.414764 12277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:33.417591 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "3e50df0d99ac409cafffdc7e4ed148e0"
format_stamp: "Formatted at 2026-09-26 07:28:33 on dist-test-slave-1dwd"
server_key: "cd78b56365d7a1518d8ac857f261c0bf"
server_key_iv: "c634abbcdeb519d6f7e29c56eb243a87"
server_key_version: "encryptionkey@0"
I20260926 07:28:33.418339 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "3e50df0d99ac409cafffdc7e4ed148e0"
format_stamp: "Formatted at 2026-09-26 07:28:33 on dist-test-slave-1dwd"
server_key: "cd78b56365d7a1518d8ac857f261c0bf"
server_key_iv: "c634abbcdeb519d6f7e29c56eb243a87"
server_key_version: "encryptionkey@0"
I20260926 07:28:33.423276 12277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:28:33.426441 12291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.427801 12277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:33.427958 12277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "3e50df0d99ac409cafffdc7e4ed148e0"
format_stamp: "Formatted at 2026-09-26 07:28:33 on dist-test-slave-1dwd"
server_key: "cd78b56365d7a1518d8ac857f261c0bf"
server_key_iv: "c634abbcdeb519d6f7e29c56eb243a87"
server_key_version: "encryptionkey@0"
I20260926 07:28:33.428164 12277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:33.449339 12277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:33.450131 12277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:33.450431 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:33.451406 12277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:33.452770 12277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:33.452854 12277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.452984 12277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:33.453100 12277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:33.485179 12277 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:46671
I20260926 07:28:33.485294 12403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:46671 every 8 connection(s)
I20260926 07:28:33.486462 12277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:33.490779 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12277
I20260926 07:28:33.490955 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:33.491456 2639 external_mini_cluster.cc:1428] Setting key e7529f494ffd8b7ba7a0e27dd84bea95
I20260926 07:28:33.498538 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.488987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:46363 (local address 127.2.147.254:39649)
0926 07:28:33.489153 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.489158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.490438 (+ 1280us) server_negotiation.cc:407] Connection header received
0926 07:28:33.492542 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.492548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.492637 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.492741 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.493776 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.494479 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.496014 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.496400 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.496531 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.497775 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:33.497779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:33.497783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:33.497798 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:33.497826 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:33.497836 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:33.497928 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:33.498141 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 07:28:33.498228 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:33.499660 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.489429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:39649 (local address 127.2.147.195:46363)
0926 07:28:33.490275 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:33.490311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:33.492235 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:33.492969 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:33.492980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:33.493070 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:33.493625 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:33.493639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.494620 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.494625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:33.495878 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:33.495889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.496609 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.496616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:33.496850 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.497466 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:33.497468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:33.497512 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:33.497589 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:33.497937 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:33.497949 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:33.497963 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:28:33.498133 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 07:28:33.498507 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":244,"threads_started":1}
I20260926 07:28:33.500936 12404 heartbeater.cc:347] Connected to a master server at 127.2.147.254:39649
I20260926 07:28:33.501235 12404 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:33.501942 12404 heartbeater.cc:511] Master 127.2.147.254:39649 requested a full tablet report, sending...
I20260926 07:28:33.503252 11957 ts_manager.cc:194] Registered new tserver with Master: 3e50df0d99ac409cafffdc7e4ed148e0 (127.2.147.195:46671)
I20260926 07:28:33.503983 11957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.195:46363
I20260926 07:28:33.506263 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:28:33.512238 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:33.510592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.2.147.254:39649)
0926 07:28:33.510765 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.510774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.511187 (+ 413us) server_negotiation.cc:407] Connection header received
0926 07:28:33.511393 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.511399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.511480 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.511578 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.511993 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43862: BlockingRecv error: recv got EOF from 127.0.0.1:43862 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:28:33.512884 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12016
I20260926 07:28:33.520478 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.513787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.2.147.193:34421)
0926 07:28:33.514275 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.514281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.514304 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:28:33.514360 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.514366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.514544 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.514708 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.515088 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.516129 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.517352 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.517888 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.518048 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.519051 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:33.519059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:33.519071 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:33.519684 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:33.519749 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:33.519768 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:33.519884 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:33.520029 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:28:33.520172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260926 07:28:33.521633 12138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:33.604295 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12016
I20260926 07:28:33.620994 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12147
I20260926 07:28:33.628702 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.621842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.2.147.194:39699)
0926 07:28:33.622310 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.622316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.622342 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:28:33.622399 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.622404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.622609 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.622776 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.623157 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.624245 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.625482 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.626039 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.626190 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.627302 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:33.627310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:33.627322 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:33.627956 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:33.628001 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:33.628019 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:33.628143 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:33.628261 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:28:33.628395 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260926 07:28:33.629822 12269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:33.709456 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12147
I20260926 07:28:33.725474 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12277
I20260926 07:28:33.737224 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:33.726541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.2.147.195:46671)
0926 07:28:33.727215 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:33.727285 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:33.727324 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:28:33.727384 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:33.727390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:33.727604 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:33.727731 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:33.728212 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.729251 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.730381 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:33.730907 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:33.731067 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:33.735833 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:33.735842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:33.735855 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:33.736443 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:33.736490 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:33.736508 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:33.736595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:33.736729 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 07:28:33.736867 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":324,"threads_started":1}
I20260926 07:28:33.738521 12399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:33.818082 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12277
I20260926 07:28:33.833824 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 11926
I20260926 07:28:33.835373 11987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:33.960295 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 11926
2026-09-26T07:28:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2354 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:34Z Disabled control of system clock
I20260926 07:28:34.028352 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45599
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:28:34.342192 12423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:34.342478 12423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:34.342572 12423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:34.349745 12423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:34.349841 12423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:34.349877 12423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:34.349937 12423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:34.349978 12423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:34.358822 12423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:45599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:34.359995 12423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:34.361667 12423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:34.373129 12428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:34.373098 12429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:34.373189 12431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:34.373827 12423 server_base.cc:1034] running on GCE node
I20260926 07:28:34.374559 12423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:34.375955 12423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:34.377146 12423 hybrid_clock.cc:648] HybridClock initialized: now 1790407714377091 us; error 51 us; skew 500 ppm
I20260926 07:28:34.377475 12423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:34.385066 12423 webserver.cc:460] Webserver started at http://127.2.147.254:39173/ using document root <none> and password file <none>
I20260926 07:28:34.385777 12423 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:34.385892 12423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:34.386222 12423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:34.389022 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "fe0fbffcd37b4adb9e9efa64a226b879"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "5ed79c1bb41d9811787e4ba7dba5a114"
server_key_iv: "e7ce5beb469edae1a324db8d8bdb6081"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.389773 12423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "fe0fbffcd37b4adb9e9efa64a226b879"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "5ed79c1bb41d9811787e4ba7dba5a114"
server_key_iv: "e7ce5beb469edae1a324db8d8bdb6081"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.394511 12423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:28:34.397689 12437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:34.398903 12423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:34.399081 12423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "fe0fbffcd37b4adb9e9efa64a226b879"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "5ed79c1bb41d9811787e4ba7dba5a114"
server_key_iv: "e7ce5beb469edae1a324db8d8bdb6081"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.399317 12423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:34.404589 12423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:34.405287 12423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:34.405580 12423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:34.423113 12423 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:45599
I20260926 07:28:34.423146 12488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:45599 every 8 connection(s)
I20260926 07:28:34.424396 12423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:34.426798 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12423
I20260926 07:28:34.427019 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:28:34.427667 2639 external_mini_cluster.cc:1428] Setting key 74fdb6319e37b23b5254618df18f8b3e
I20260926 07:28:34.428130 12489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:34.435947 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:34.429617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.2.147.254:45599)
0926 07:28:34.430611 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:34.430622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:34.430664 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:28:34.432553 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:34.432578 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:34.432983 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:34.433300 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:34.434367 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:34.434373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:34.434385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:34.434401 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:34.434459 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:34.434479 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:34.434643 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:34.434754 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:28:34.435043 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":363,"threads_started":1}
I20260926 07:28:34.440752 12489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879: Bootstrap starting.
I20260926 07:28:34.443821 12489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:34.444798 12489 log.cc:824] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:34.447396 12489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879: No bootstrap required, opened a new log
I20260926 07:28:34.454546 12489 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } }
I20260926 07:28:34.454898 12489 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:34.455000 12489 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0fbffcd37b4adb9e9efa64a226b879, State: Initialized, Role: FOLLOWER
I20260926 07:28:34.455577 12489 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } }
I20260926 07:28:34.455811 12489 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:34.455947 12489 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:34.456125 12489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:34.458102 12489 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } }
I20260926 07:28:34.458573 12489 leader_election.cc:302] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe0fbffcd37b4adb9e9efa64a226b879; no voters:
I20260926 07:28:34.459158 12489 leader_election.cc:288] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:34.459301 12494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:34.460031 12494 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [term 1 LEADER]: Becoming Leader. State: Replica: fe0fbffcd37b4adb9e9efa64a226b879, State: Running, Role: LEADER
I20260926 07:28:34.460570 12494 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } }
I20260926 07:28:34.461355 12489 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:34.463582 12495 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } } }
I20260926 07:28:34.463496 12496 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe0fbffcd37b4adb9e9efa64a226b879. Latest consensus state: current_term: 1 leader_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0fbffcd37b4adb9e9efa64a226b879" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 45599 } } }
I20260926 07:28:34.463933 12495 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:34.463963 12496 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879 [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:34.467267 12502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:34.471251 12502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:34.478991 12502 catalog_manager.cc:1347] Generated new cluster ID: 77e5d13f74e4495788d46020622afb26
I20260926 07:28:34.479107 12502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:34.509999 12502 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:34.510424 12502 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:34.517041 12502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe0fbffcd37b4adb9e9efa64a226b879: Generated new TSK 0
I20260926 07:28:34.517982 12502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:28:34.534518 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--builtin_ntp_servers=127.2.147.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:28:34.855613 12513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:34.855906 12513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:34.855993 12513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:34.863139 12513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:34.863250 12513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:34.863353 12513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:34.872006 12513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:34.873263 12513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:34.874923 12513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:34.887734 12519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:34.887768 12518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:34.887797 12521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:34.888531 12513 server_base.cc:1034] running on GCE node
I20260926 07:28:34.889097 12513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:34.890034 12513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:34.891294 12513 hybrid_clock.cc:648] HybridClock initialized: now 1790407714891237 us; error 56 us; skew 500 ppm
I20260926 07:28:34.891620 12513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:34.893743 12513 webserver.cc:460] Webserver started at http://127.2.147.193:40921/ using document root <none> and password file <none>
I20260926 07:28:34.894424 12513 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:34.894549 12513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:34.894886 12513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:34.897817 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "b9957e35b76d43d5b6a87f02df514e72"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "4fe2f2d926c3e25f0d766288b8f99ad2"
server_key_iv: "105df2817d18f59d1f2ffb44dc801b2a"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.898607 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "b9957e35b76d43d5b6a87f02df514e72"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "4fe2f2d926c3e25f0d766288b8f99ad2"
server_key_iv: "105df2817d18f59d1f2ffb44dc801b2a"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.903864 12513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 07:28:34.907104 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:34.908455 12513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:34.908618 12513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "b9957e35b76d43d5b6a87f02df514e72"
format_stamp: "Formatted at 2026-09-26 07:28:34 on dist-test-slave-1dwd"
server_key: "4fe2f2d926c3e25f0d766288b8f99ad2"
server_key_iv: "105df2817d18f59d1f2ffb44dc801b2a"
server_key_version: "encryptionkey@0"
I20260926 07:28:34.908839 12513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:34.914561 12513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:34.915431 12513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:34.915691 12513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:34.916550 12513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:34.917893 12513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:34.918000 12513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:34.918120 12513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:34.918227 12513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:34.949678 12513 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:34229
I20260926 07:28:34.949731 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:34229 every 8 connection(s)
I20260926 07:28:34.950982 12513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:34.956550 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12513
I20260926 07:28:34.956749 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:34.957372 2639 external_mini_cluster.cc:1428] Setting key 65c8d8f30ce9c875275c48a292d3b0f8
I20260926 07:28:34.959149 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:34.953553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:53649 (local address 127.2.147.254:45599)
0926 07:28:34.953994 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:34.953998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:34.955102 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 07:28:34.956783 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:34.956789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:34.956868 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:34.956987 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:34.958374 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:34.958380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:34.958385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:34.958402 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:34.958450 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:34.958462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:34.958582 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:34.958768 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 07:28:34.958831 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":251,"threads_started":1}
I20260926 07:28:34.960016 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--builtin_ntp_servers=127.2.147.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:28:34.960072 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:34.954021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45599 (local address 127.2.147.193:53649)
0926 07:28:34.954935 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:34.954972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:34.956511 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:34.957239 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:34.957247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:34.957339 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:34.958069 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:34.958072 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:34.958124 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:34.958231 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:34.958557 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:34.958567 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:34.958574 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:28:34.958852 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 07:28:34.959125 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":280,"threads_started":1}
I20260926 07:28:34.963124 12640 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45599
I20260926 07:28:34.963428 12640 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:34.963979 12640 heartbeater.cc:511] Master 127.2.147.254:45599 requested a full tablet report, sending...
I20260926 07:28:34.966127 12454 ts_manager.cc:194] Registered new tserver with Master: b9957e35b76d43d5b6a87f02df514e72 (127.2.147.193:34229)
W20260926 07:28:35.280418 12644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:35.280733 12644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:35.280896 12644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:35.288197 12644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:35.288329 12644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:35.288513 12644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:35.297472 12644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:35.298743 12644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:35.300563 12644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:35.313539 12650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:35.313599 12652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:35.313710 12649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:35.314779 12644 server_base.cc:1034] running on GCE node
I20260926 07:28:35.315376 12644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:35.316229 12644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:35.317430 12644 hybrid_clock.cc:648] HybridClock initialized: now 1790407715317380 us; error 40 us; skew 500 ppm
I20260926 07:28:35.317695 12644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:35.319789 12644 webserver.cc:460] Webserver started at http://127.2.147.194:34401/ using document root <none> and password file <none>
I20260926 07:28:35.320458 12644 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:35.320540 12644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:35.320802 12644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:35.325131 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "80437d0e6ec848cb9cb2c2d31cd830c6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "6911738f51c8a40fb9d5bd8176249eb1"
server_key_iv: "59d3da2a58afef2135737dfc30365828"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.325861 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "80437d0e6ec848cb9cb2c2d31cd830c6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "6911738f51c8a40fb9d5bd8176249eb1"
server_key_iv: "59d3da2a58afef2135737dfc30365828"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.330699 12644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:28:35.334080 12658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.335441 12644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:35.335564 12644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "80437d0e6ec848cb9cb2c2d31cd830c6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "6911738f51c8a40fb9d5bd8176249eb1"
server_key_iv: "59d3da2a58afef2135737dfc30365828"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.335732 12644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:35.341022 12644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:35.341789 12644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:35.342006 12644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:35.342787 12644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:35.344094 12644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:35.344177 12644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.344252 12644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:35.344306 12644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.376509 12644 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:39305
I20260926 07:28:35.376585 12770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:39305 every 8 connection(s)
I20260926 07:28:35.377807 12644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:35.382350 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12644
I20260926 07:28:35.382532 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:35.382968 2639 external_mini_cluster.cc:1428] Setting key 433b59a57be28e2593ff97ab5c0eb49b
I20260926 07:28:35.385524 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--builtin_ntp_servers=127.2.147.212:40605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:28:35.385893 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.380306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:33079 (local address 127.2.147.254:45599)
0926 07:28:35.380488 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:35.380492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:35.381767 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 07:28:35.383639 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:35.383644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:35.383728 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:35.383838 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:35.385143 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:35.385148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:35.385153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:35.385171 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:35.385209 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:35.385220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:35.385338 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:35.385527 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:28:35.385590 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:28:35.386878 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.380772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45599 (local address 127.2.147.194:33079)
0926 07:28:35.381619 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:35.381659 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:35.383362 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:35.384076 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:35.384084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:35.384173 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:35.384855 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:35.384857 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:35.384894 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:35.384980 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:35.385313 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:35.385323 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:35.385330 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:28:35.385608 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 07:28:35.385895 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":254,"threads_started":1}
I20260926 07:28:35.388479 12771 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45599
I20260926 07:28:35.388761 12771 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:35.389300 12771 heartbeater.cc:511] Master 127.2.147.254:45599 requested a full tablet report, sending...
I20260926 07:28:35.390666 12454 ts_manager.cc:194] Registered new tserver with Master: 80437d0e6ec848cb9cb2c2d31cd830c6 (127.2.147.194:39305)
W20260926 07:28:35.714707 12774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:35.714972 12774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:35.715057 12774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:35.722172 12774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:35.722260 12774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:35.722362 12774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:35.731192 12774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:40605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:45599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:35.732615 12774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:35.734275 12774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:35.747316 12779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:35.747345 12782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:35.747381 12780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:35.748625 12774 server_base.cc:1034] running on GCE node
I20260926 07:28:35.749125 12774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:35.749966 12774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:35.751152 12774 hybrid_clock.cc:648] HybridClock initialized: now 1790407715751095 us; error 56 us; skew 500 ppm
I20260926 07:28:35.751492 12774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:28:35.753563 12774 webserver.cc:460] Webserver started at http://127.2.147.195:43519/ using document root <none> and password file <none>
I20260926 07:28:35.754215 12774 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:35.754325 12774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:35.754652 12774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:35.757393 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "1984741722b045bfbb4ac1c4189b3cb6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "7721b8bc9dd31480b451375ef8cdc325"
server_key_iv: "b63c9e70da403a216c4add3d568a6fae"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.758149 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "1984741722b045bfbb4ac1c4189b3cb6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "7721b8bc9dd31480b451375ef8cdc325"
server_key_iv: "b63c9e70da403a216c4add3d568a6fae"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.764005 12774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 07:28:35.767429 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.768728 12774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:28:35.768900 12774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "1984741722b045bfbb4ac1c4189b3cb6"
format_stamp: "Formatted at 2026-09-26 07:28:35 on dist-test-slave-1dwd"
server_key: "7721b8bc9dd31480b451375ef8cdc325"
server_key_iv: "b63c9e70da403a216c4add3d568a6fae"
server_key_version: "encryptionkey@0"
I20260926 07:28:35.769121 12774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:35.774443 12774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:35.775177 12774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:35.775507 12774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:35.776350 12774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:35.777714 12774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:35.777829 12774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.777964 12774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:35.778051 12774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:35.809662 12774 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:37773
I20260926 07:28:35.809715 12900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:37773 every 8 connection(s)
I20260926 07:28:35.810994 12774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:35.818243 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12774
I20260926 07:28:35.818441 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:35.818861 2639 external_mini_cluster.cc:1428] Setting key 5d0b9296b7f93eaa9e7b1d74d2e7e90f
I20260926 07:28:35.818887 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.813586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:53573 (local address 127.2.147.254:45599)
0926 07:28:35.813780 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:35.813786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:35.815185 (+ 1399us) server_negotiation.cc:407] Connection header received
0926 07:28:35.816824 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:35.816828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:35.816878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:35.816965 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:35.818167 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:35.818172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:35.818177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:35.818194 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:35.818231 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:35.818242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:35.818371 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:35.818549 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:28:35.818616 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:28:35.820076 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.814166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:45599 (local address 127.2.147.195:53573)
0926 07:28:35.815039 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:35.815074 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:35.816598 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:35.817171 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:35.817179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:35.817268 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:35.817877 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:28:35.817880 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:35.817915 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:28:35.818003 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:35.818334 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:35.818342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:35.818351 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:28:35.818630 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 07:28:35.818870 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
I20260926 07:28:35.821269 12901 heartbeater.cc:347] Connected to a master server at 127.2.147.254:45599
I20260926 07:28:35.821580 12901 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:35.822252 12901 heartbeater.cc:511] Master 127.2.147.254:45599 requested a full tablet report, sending...
I20260926 07:28:35.823498 12453 ts_manager.cc:194] Registered new tserver with Master: 1984741722b045bfbb4ac1c4189b3cb6 (127.2.147.195:37773)
I20260926 07:28:35.833843 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:28:35.839361 12642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:35.837889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.2.147.254:45599)
0926 07:28:35.838094 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:35.838103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:35.838528 (+ 425us) server_negotiation.cc:407] Connection header received
0926 07:28:35.838649 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:35.838653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:35.838701 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:35.838773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:35.839128 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41554: BlockingRecv error: recv got EOF from 127.0.0.1:41554 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:28:35.840080 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12513
I20260926 07:28:35.843524 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.841157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.2.147.193:34229)
0926 07:28:35.841651 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:35.841657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:35.841682 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:28:35.841734 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:35.841739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:35.841931 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:35.842051 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:35.842898 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:35.842904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:35.842915 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:35.842933 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:35.842973 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:35.842987 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:35.843086 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:35.843166 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 07:28:35.843300 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260926 07:28:35.844775 12635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:35.924387 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12513
I20260926 07:28:35.940701 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12644
I20260926 07:28:35.944270 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:35.941660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.2.147.194:39305)
0926 07:28:35.942175 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:35.942183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:35.942207 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:28:35.942266 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:35.942271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:35.942511 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:35.942629 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:35.943585 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:35.943591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:35.943602 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:35.943619 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:35.943656 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:35.943670 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:35.943768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:35.943876 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:28:35.943998 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260926 07:28:35.945458 12766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:36.024706 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12644
I20260926 07:28:36.040361 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12774
I20260926 07:28:36.044142 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:36.041260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.2.147.195:37773)
0926 07:28:36.041799 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:36.041813 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:36.041842 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:28:36.041901 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:36.041907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:36.042117 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:36.042288 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:36.043418 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:36.043425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:36.043436 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:28:36.043448 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:36.043497 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:28:36.043510 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:36.043687 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:36.043752 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:28:36.043853 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
I20260926 07:28:36.045296 12896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:36.125250 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12774
I20260926 07:28:36.140558 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12423
I20260926 07:28:36.141659 12484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:36.240772 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12423
2026-09-26T07:28:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2281 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:36 dist-test-slave-1dwd krb5kdc[12919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:36 dist-test-slave-1dwd krb5kdc[12919](info): set up 2 sockets
Sep 26 07:28:36 dist-test-slave-1dwd krb5kdc[12919](info): commencing operation
krb5kdc: starting...
W20260926 07:28:37.340237 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:28:37 dist-test-slave-1dwd krb5kdc[12919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:28:37.551790 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43985
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:37.866123 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:37.866413 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:37.866518 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:37.873785 12932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:37.873889 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:37.873934 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:37.873997 12932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:37.874034 12932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:37.883678 12932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:37.884840 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:37.886085 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:37.897307 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:37.897330 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:37.897665 12940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:37.898532 12932 server_base.cc:1034] running on GCE node
I20260926 07:28:37.899359 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:37.900700 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:37.901901 12932 hybrid_clock.cc:648] HybridClock initialized: now 1790407717901860 us; error 34 us; skew 500 ppm
I20260926 07:28:37.902236 12932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:37 dist-test-slave-1dwd krb5kdc[12919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:37.905328 12932 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:28:37.906821 12932 webserver.cc:460] Webserver started at http://127.2.147.254:43931/ using document root <none> and password file <none>
I20260926 07:28:37.907545 12932 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:37.907681 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:37.907979 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:37.911588 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "0ab077aa793d443089120e119f07c23f"
format_stamp: "Formatted at 2026-09-26 07:28:37 on dist-test-slave-1dwd"
server_key: "3a8fbe17f0368d1e91a867ab7caebb3d"
server_key_iv: "48b48ae73f6b1c942668c2ff0f13a412"
server_key_version: "encryptionkey@0"
I20260926 07:28:37.912537 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "0ab077aa793d443089120e119f07c23f"
format_stamp: "Formatted at 2026-09-26 07:28:37 on dist-test-slave-1dwd"
server_key: "3a8fbe17f0368d1e91a867ab7caebb3d"
server_key_iv: "48b48ae73f6b1c942668c2ff0f13a412"
server_key_version: "encryptionkey@0"
I20260926 07:28:37.918147 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:28:37.921476 12947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:37.922736 12932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:28:37.922914 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "0ab077aa793d443089120e119f07c23f"
format_stamp: "Formatted at 2026-09-26 07:28:37 on dist-test-slave-1dwd"
server_key: "3a8fbe17f0368d1e91a867ab7caebb3d"
server_key_iv: "48b48ae73f6b1c942668c2ff0f13a412"
server_key_version: "encryptionkey@0"
I20260926 07:28:37.923098 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:37.942036 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:37.945734 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:37.946003 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:37.963663 12932 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:43985
I20260926 07:28:37.963706 12998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:43985 every 8 connection(s)
I20260926 07:28:37.964959 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:37.968406 12999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:37.972492 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 12932
I20260926 07:28:37.972715 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:28:37.973490 2639 external_mini_cluster.cc:1428] Setting key 10a5943dda1ca734bb824d8156849117
I20260926 07:28:37.979732 12999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: Bootstrap starting.
I20260926 07:28:37.983183 12999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:37.984280 12999 log.cc:824] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:37.986827 12999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: No bootstrap required, opened a new log
Sep 26 07:28:37 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:37.996865 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:37.975486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.2.147.254:43985)
0926 07:28:37.976236 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:37.976246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:37.976283 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:28:37.978282 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:37.978302 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:37.978657 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:37.978961 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:37.980799 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:37.982098 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:37.983865 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:37.984542 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:37.984662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:37.988601 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:37.988611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:37.988634 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:37.988679 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:37.992612 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:37.993235 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:37.993243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:37.993251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:37.993437 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:37.993769 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:37.993773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:37.993777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:37.994462 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:37.994736 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:37.995023 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 07:28:37.995475 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":376,"threads_started":1}
I20260926 07:28:37.997498 12999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } }
I20260926 07:28:37.997887 12999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:37.998003 12999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab077aa793d443089120e119f07c23f, State: Initialized, Role: FOLLOWER
I20260926 07:28:37.998592 12999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } }
I20260926 07:28:37.998816 12999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:37.998950 12999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:37.999136 12999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:38.001518 12999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } }
I20260926 07:28:38.001996 12999 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab077aa793d443089120e119f07c23f; no voters:
I20260926 07:28:38.002573 12999 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:38.002729 13004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:38.004480 12999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:38.004832 13004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [term 1 LEADER]: Becoming Leader. State: Replica: 0ab077aa793d443089120e119f07c23f, State: Running, Role: LEADER
I20260926 07:28:38.005353 13004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } }
I20260926 07:28:38.008543 13006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab077aa793d443089120e119f07c23f. Latest consensus state: current_term: 1 leader_uuid: "0ab077aa793d443089120e119f07c23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } } }
I20260926 07:28:38.008520 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab077aa793d443089120e119f07c23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab077aa793d443089120e119f07c23f" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 43985 } } }
I20260926 07:28:38.008912 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:38.008912 13006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:38.010528 13014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:38.014711 13014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:38.022964 13014 catalog_manager.cc:1347] Generated new cluster ID: 954fbd5471c94530a77e5ce355303eed
I20260926 07:28:38.023066 13014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:38.037393 13014 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:38.038700 13014 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:38.045795 13014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: Generated new TSK 0
I20260926 07:28:38.046684 13014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:28:38.100553 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:38.426404 13025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:38.426672 13025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:38.426847 13025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:38.434240 13025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:38.434355 13025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:38.434507 13025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:28:38.444473 13025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:38.445782 13025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:38.447108 13025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:38.460769 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:38.460681 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:38.460683 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:38.461575 13025 server_base.cc:1034] running on GCE node
I20260926 07:28:38.462110 13025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:38.462975 13025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:38.464203 13025 hybrid_clock.cc:648] HybridClock initialized: now 1790407718464132 us; error 65 us; skew 500 ppm
I20260926 07:28:38.464529 13025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:38 dist-test-slave-1dwd krb5kdc[12919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:38.467674 13025 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:28:38.469080 13025 webserver.cc:460] Webserver started at http://127.2.147.193:45265/ using document root <none> and password file <none>
I20260926 07:28:38.469763 13025 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:38.469909 13025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:38.470212 13025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:38.473114 13025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "a5ede1b73d994a75b0eb5fac4327d62e"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "ff133348f0cc7620d99a41d71f3e6955"
server_key_iv: "d99f169dcee56349aa77ee3f18c20458"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.473862 13025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "a5ede1b73d994a75b0eb5fac4327d62e"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "ff133348f0cc7620d99a41d71f3e6955"
server_key_iv: "d99f169dcee56349aa77ee3f18c20458"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.478820 13025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 07:28:38.482158 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:38.483518 13025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:28:38.483692 13025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "a5ede1b73d994a75b0eb5fac4327d62e"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "ff133348f0cc7620d99a41d71f3e6955"
server_key_iv: "d99f169dcee56349aa77ee3f18c20458"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.483896 13025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:38.496678 13025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:38.500978 13025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:38.501291 13025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:38.502199 13025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:38.503545 13025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:38.503646 13025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:38.503799 13025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:38.503876 13025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:38.536631 13025 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33975
I20260926 07:28:38.536677 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33975 every 8 connection(s)
I20260926 07:28:38.537916 13025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:38.542551 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 13025
I20260926 07:28:38.542775 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:38.543596 2639 external_mini_cluster.cc:1428] Setting key d5391962dae65c0af3b06bfd3514437f
Sep 26 07:28:38 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:38.558048 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:38.540565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:38427 (local address 127.2.147.254:43985)
0926 07:28:38.540985 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:38.540989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:38.541838 (+ 849us) server_negotiation.cc:407] Connection header received
0926 07:28:38.544417 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:38.544423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:38.544517 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:38.544635 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:38.546291 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:38.546978 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:38.548679 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:38.549076 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:38.549211 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:38.552563 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:38.552568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:38.552571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:38.552601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:38.555078 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:38.555815 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:38.555823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:38.555831 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:38.555948 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:38.556389 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:38.556401 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:38.556403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:38.556645 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:38.556798 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:38.557472 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 07:28:38.557683 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":239,"threads_started":1}
I20260926 07:28:38.558986 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:38.541009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43985 (local address 127.2.147.193:38427)
0926 07:28:38.541683 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:38.541720 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:38.544128 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:38.544865 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:38.544874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:38.545412 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:38.546132 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:38.546151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:38.547120 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:38.547125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:38.548534 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:38.548545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:38.549291 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:38.549298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:38.549527 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:38.550221 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:38.550241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:38.552354 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:38.555259 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:38.555270 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:38.555290 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:38.555690 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:38.556071 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:38.556076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:38.556079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:38.556271 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:38.556771 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:38.556779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:38.557115 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 07:28:38.557370 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 07:28:38.557897 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260926 07:28:38.560920 13153 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43985
I20260926 07:28:38.561352 13153 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:38.562256 13153 heartbeater.cc:511] Master 127.2.147.254:43985 requested a full tablet report, sending...
I20260926 07:28:38.564735 12964 ts_manager.cc:194] Registered new tserver with Master: a5ede1b73d994a75b0eb5fac4327d62e (127.2.147.193:33975)
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:28:38.566383 12964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:38427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:28:38.592092 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:38.908620 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:38.908924 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:38.909044 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:38.916018 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:38.916111 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:38.916217 13159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:28:38.925663 13159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:38.926903 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:38.928505 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:38.941802 13164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:38.941871 13167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:38.941902 13165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:38.942392 13159 server_base.cc:1034] running on GCE node
I20260926 07:28:38.943097 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:38.944000 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:38.945221 13159 hybrid_clock.cc:648] HybridClock initialized: now 1790407718945174 us; error 57 us; skew 500 ppm
I20260926 07:28:38.945526 13159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:38 dist-test-slave-1dwd krb5kdc[12919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:38.948520 13159 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:28:38.949846 13159 webserver.cc:460] Webserver started at http://127.2.147.194:38997/ using document root <none> and password file <none>
I20260926 07:28:38.950510 13159 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:38.950652 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:38.950950 13159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:38.953785 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "e05e00a01f7e4ca0953833dcd2c722ec"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "b607f8486caaa39d64603052a3ca1e6d"
server_key_iv: "52e95214605f164ecb78458d737ab590"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.954519 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "e05e00a01f7e4ca0953833dcd2c722ec"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "b607f8486caaa39d64603052a3ca1e6d"
server_key_iv: "52e95214605f164ecb78458d737ab590"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.959386 13159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 07:28:38.962654 13174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:38.964027 13159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:38.964180 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "e05e00a01f7e4ca0953833dcd2c722ec"
format_stamp: "Formatted at 2026-09-26 07:28:38 on dist-test-slave-1dwd"
server_key: "b607f8486caaa39d64603052a3ca1e6d"
server_key_iv: "52e95214605f164ecb78458d737ab590"
server_key_version: "encryptionkey@0"
I20260926 07:28:38.964357 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:38.989593 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:38.993328 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:38.993597 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:38.994464 13159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:38.995747 13159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:38.995820 13159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:38.995978 13159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:38.996028 13159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:39.027493 13159 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:46307
I20260926 07:28:39.027514 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:46307 every 8 connection(s)
I20260926 07:28:39.028760 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:39.032766 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 13159
I20260926 07:28:39.032969 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:39.033509 2639 external_mini_cluster.cc:1428] Setting key 9c2dd262468089b74e4a1a7889e03447
Sep 26 07:28:39 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407718, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:39.050786 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.031187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:47125 (local address 127.2.147.254:43985)
0926 07:28:39.031410 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.031416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.032733 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 07:28:39.034996 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.035002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.035074 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.035176 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:39.037182 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.038060 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.040486 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.040896 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.040989 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.045254 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:39.045259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:39.045262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:39.045292 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:39.047869 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.048582 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.048595 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.048597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.048721 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.049153 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.049164 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.049167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.049410 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:39.049568 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.050224 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.050438 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:28:39.051857 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.031768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43985 (local address 127.2.147.194:47125)
0926 07:28:39.032531 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.032592 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.034735 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.035445 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.035479 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.036173 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:39.036992 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.037019 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.038214 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.038222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.040330 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.040347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.041117 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.041126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.041451 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.042181 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:39.042213 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:39.045032 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:39.048017 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:39.048026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:39.048043 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:39.048457 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:39.048844 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:39.048850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:39.048854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:39.049035 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:39.049544 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:39.049551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:39.049877 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.050139 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.050674 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":260,"threads_started":1}
I20260926 07:28:39.053531 13287 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43985
I20260926 07:28:39.053839 13287 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:39.054553 13287 heartbeater.cc:511] Master 127.2.147.254:43985 requested a full tablet report, sending...
I20260926 07:28:39.056075 12964 ts_manager.cc:194] Registered new tserver with Master: e05e00a01f7e4ca0953833dcd2c722ec (127.2.147.194:46307)
I20260926 07:28:39.057272 12964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:47125
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:28:39.087800 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:39.438798 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:39.439052 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:39.439141 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:39.446270 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:39.446368 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:39.446513 13292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:28:39.456228 13292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:43985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:28:39.457477 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:39.458755 13292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:39.471899 13297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:39.471944 13300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:39.471899 13298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:39.472785 13292 server_base.cc:1034] running on GCE node
I20260926 07:28:39.473367 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:39.474272 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:39.475467 13292 hybrid_clock.cc:648] HybridClock initialized: now 1790407719475425 us; error 34 us; skew 500 ppm
I20260926 07:28:39.475764 13292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:28:39 dist-test-slave-1dwd krb5kdc[12919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:39.482122 13292 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:28:39.483840 13292 webserver.cc:460] Webserver started at http://127.2.147.195:32807/ using document root <none> and password file <none>
I20260926 07:28:39.484542 13292 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:39.484686 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:39.484978 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:39.487874 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "b375bd488e7a43f9958b2e4ffb219e96"
format_stamp: "Formatted at 2026-09-26 07:28:39 on dist-test-slave-1dwd"
server_key: "2316824594534a1b83dff1ace8b9276d"
server_key_iv: "b1d28994a42b60ac2349d26eeed6b74b"
server_key_version: "encryptionkey@0"
I20260926 07:28:39.488672 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "b375bd488e7a43f9958b2e4ffb219e96"
format_stamp: "Formatted at 2026-09-26 07:28:39 on dist-test-slave-1dwd"
server_key: "2316824594534a1b83dff1ace8b9276d"
server_key_iv: "b1d28994a42b60ac2349d26eeed6b74b"
server_key_version: "encryptionkey@0"
I20260926 07:28:39.493569 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:28:39.496788 13307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:39.497995 13292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:28:39.498165 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "b375bd488e7a43f9958b2e4ffb219e96"
format_stamp: "Formatted at 2026-09-26 07:28:39 on dist-test-slave-1dwd"
server_key: "2316824594534a1b83dff1ace8b9276d"
server_key_iv: "b1d28994a42b60ac2349d26eeed6b74b"
server_key_version: "encryptionkey@0"
I20260926 07:28:39.498368 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:39.511721 13292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:39.515380 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:39.515641 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:39.516486 13292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:39.517710 13292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:39.517820 13292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:39.517966 13292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:39.518040 13292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:39.550521 13292 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:46557
I20260926 07:28:39.550561 13419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:46557 every 8 connection(s)
I20260926 07:28:39.551787 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:39.554030 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 13292
I20260926 07:28:39.554230 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:39.554737 2639 external_mini_cluster.cc:1428] Setting key 093ca86fbe796031a9f5db86c2930d47
Sep 26 07:28:39 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407719, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:28:39.569314 13153 heartbeater.cc:503] Master 127.2.147.254:43985 was elected leader, sending a full tablet report...
I20260926 07:28:39.572496 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.554355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:51871 (local address 127.2.147.254:43985)
0926 07:28:39.554835 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.554840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.558210 (+ 3370us) server_negotiation.cc:407] Connection header received
0926 07:28:39.558279 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.558283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.558340 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.558419 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:39.559847 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.560577 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.561987 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.562371 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.562467 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.566419 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:39.566427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:39.566435 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:39.566474 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:39.569431 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.570356 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.570359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.570361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.570478 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.570973 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.570980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.570986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.571347 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:39.571459 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.572076 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.572236 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260926 07:28:39.573428 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.554996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:43985 (local address 127.2.147.195:51871)
0926 07:28:39.555837 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.555889 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.557961 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.558629 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.558636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.559103 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:39.559711 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.559728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.560695 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.560698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.561872 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.561883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.562537 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.562541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.562744 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.563639 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:39.563667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:39.566222 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:39.569599 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:39.569628 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:39.569643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:39.570228 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:39.570590 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:39.570595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:39.570598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:39.570839 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:39.571454 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:39.571460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:39.571762 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.571982 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.572354 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260926 07:28:39.574676 13420 heartbeater.cc:347] Connected to a master server at 127.2.147.254:43985
I20260926 07:28:39.574922 13420 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:39.575625 13420 heartbeater.cc:511] Master 127.2.147.254:43985 requested a full tablet report, sending...
I20260926 07:28:39.576840 12964 ts_manager.cc:194] Registered new tserver with Master: b375bd488e7a43f9958b2e4ffb219e96 (127.2.147.195:46557)
I20260926 07:28:39.577616 12964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:51871
I20260926 07:28:39.581972 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:28:39.597280 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.586222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.2.147.254:43985)
0926 07:28:39.586372 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.586377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.586859 (+ 482us) server_negotiation.cc:407] Connection header received
0926 07:28:39.587012 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.587016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.587089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.587167 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:39.588621 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.589329 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.590471 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.590858 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.590945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.593000 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:39.593005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:39.593008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:39.593034 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:39.595343 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.595881 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.595884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.595886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.595991 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.596269 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.596272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.596274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.596533 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:39.596692 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.596939 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.597082 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:28:39.601064 12964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:39.608407 12964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:39.630291 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.619620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33975 (local address 127.0.0.1:39196)
0926 07:28:39.620280 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.620338 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.620513 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.621742 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.621752 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.621794 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:39.622280 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.622294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.624244 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.624249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.627592 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.627611 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.629533 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.629539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.629697 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.629724 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.629800 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.629898 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":861,"thread_start_us":285,"threads_started":1}
I20260926 07:28:39.630479 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.619992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.2.147.193:33975)
0926 07:28:39.620795 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.620806 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.620846 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:28:39.620952 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.620959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.621228 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.621449 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:39.622516 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.624082 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.627766 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.629326 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.629475 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.629719 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.629869 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.630049 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":1148,"threads_started":1}
I20260926 07:28:39.630971 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.621013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.2.147.194:46307)
0926 07:28:39.621894 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.621903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.622438 (+ 535us) server_negotiation.cc:407] Connection header received
0926 07:28:39.622533 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.622541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.622769 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.622947 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:39.623793 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.625299 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.627867 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.628991 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.629154 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.629371 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.630296 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.630715 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":527,"threads_started":1}
I20260926 07:28:39.631258 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.620741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:46307 (local address 127.0.0.1:48186)
0926 07:28:39.622165 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.622183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.622298 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.623183 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.623187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.623233 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:39.623621 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.623632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.625468 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.625473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.627711 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.627721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.629209 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.629215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.629443 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.629517 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.629637 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.629832 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":37,"thread_start_us":1117,"threads_started":1}
I20260926 07:28:39.634220 13088 tablet_service.cc:1467] Processing CreateTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d (DEFAULT_TABLE table=test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:39.634649 13222 tablet_service.cc:1467] Processing CreateTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d (DEFAULT_TABLE table=test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:39.635782 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a647ef71f1a84a079f8c5b4f7dcae26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:39.635943 13222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a647ef71f1a84a079f8c5b4f7dcae26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:39.636559 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.622285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:46557 (local address 127.0.0.1:42892)
0926 07:28:39.627876 (+ 5591us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.627895 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.628003 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.628939 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.628944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.628968 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:39.629390 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.629405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.631907 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.631912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.633926 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.633940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.635711 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.635718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.635960 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.636002 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.636093 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.636221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5453,"thread_start_us":5042,"threads_started":1}
I20260926 07:28:39.637253 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.622627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.2.147.195:46557)
0926 07:28:39.627127 (+ 4500us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.627136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.628130 (+ 994us) server_negotiation.cc:407] Connection header received
0926 07:28:39.628248 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.628256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.628500 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.628690 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:39.629609 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.631743 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.634099 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.635509 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.635643 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.635867 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.636186 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.636342 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4318,"thread_start_us":4167,"threads_started":1}
I20260926 07:28:39.641007 13355 tablet_service.cc:1467] Processing CreateTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d (DEFAULT_TABLE table=test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:39.642481 13355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a647ef71f1a84a079f8c5b4f7dcae26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:39.647113 13439 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Bootstrap starting.
I20260926 07:28:39.649905 13440 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Bootstrap starting.
I20260926 07:28:39.651592 13439 tablet_bootstrap.cc:654] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:39.652788 13440 tablet_bootstrap.cc:654] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:39.652875 13439 log.cc:824] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:39.653971 13440 log.cc:824] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:39.656147 13439 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: No bootstrap required, opened a new log
I20260926 07:28:39.656621 13440 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: No bootstrap required, opened a new log
I20260926 07:28:39.656719 13443 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Bootstrap starting.
I20260926 07:28:39.656875 13440 ts_tablet_manager.cc:1397] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 07:28:39.656921 13439 ts_tablet_manager.cc:1397] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 07:28:39.660527 13443 tablet_bootstrap.cc:654] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:39.661720 13443 log.cc:824] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:39.664789 13443 tablet_bootstrap.cc:492] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: No bootstrap required, opened a new log
I20260926 07:28:39.665055 13443 ts_tablet_manager.cc:1397] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:28:39.664784 13439 raft_consensus.cc:348] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.665206 13439 raft_consensus.cc:374] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:39.665349 13439 raft_consensus.cc:729] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ede1b73d994a75b0eb5fac4327d62e, State: Initialized, Role: FOLLOWER
I20260926 07:28:39.666101 13439 consensus_queue.cc:260] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.666968 13440 raft_consensus.cc:348] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.667397 13440 raft_consensus.cc:374] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:39.667487 13440 raft_consensus.cc:729] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05e00a01f7e4ca0953833dcd2c722ec, State: Initialized, Role: FOLLOWER
I20260926 07:28:39.667968 13439 ts_tablet_manager.cc:1428] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 07:28:39.668124 13440 consensus_queue.cc:260] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.669252 13287 heartbeater.cc:503] Master 127.2.147.254:43985 was elected leader, sending a full tablet report...
I20260926 07:28:39.669925 13440 ts_tablet_manager.cc:1428] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 07:28:39.674995 13443 raft_consensus.cc:348] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.675428 13443 raft_consensus.cc:374] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:39.675561 13443 raft_consensus.cc:729] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b375bd488e7a43f9958b2e4ffb219e96, State: Initialized, Role: FOLLOWER
I20260926 07:28:39.676105 13443 consensus_queue.cc:260] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.677023 13420 heartbeater.cc:503] Master 127.2.147.254:43985 was elected leader, sending a full tablet report...
I20260926 07:28:39.677421 13443 ts_tablet_manager.cc:1428] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 07:28:39.688117 13447 raft_consensus.cc:482] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:39.688360 13447 raft_consensus.cc:504] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.690003 13447 leader_election.cc:288] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ede1b73d994a75b0eb5fac4327d62e (127.2.147.193:33975), e05e00a01f7e4ca0953833dcd2c722ec (127.2.147.194:46307)
I20260926 07:28:39.696563 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.690229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33975 (local address 127.2.147.195:41499)
0926 07:28:39.690428 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.690446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.690576 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.691079 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.691085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.691118 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:39.691355 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.691370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.692632 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.692636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.694613 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.694627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.695887 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.695893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.696081 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.696114 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.696203 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.696335 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":71}
I20260926 07:28:39.696869 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.690333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:41499 (local address 127.2.147.193:33975)
0926 07:28:39.690526 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.690531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.690549 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:28:39.690717 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.690725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.690841 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.690955 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:39.691499 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.692500 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.694762 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.695738 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.695866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.695917 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.696245 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.696378 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:39.697010 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.690717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:51183 (local address 127.2.147.194:46307)
0926 07:28:39.690983 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.690989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.691405 (+ 416us) server_negotiation.cc:407] Connection header received
0926 07:28:39.691538 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.691544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.691619 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.691711 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:39.692148 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.693006 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.695173 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.696099 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.696252 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.696327 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.696664 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.696802 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 07:28:39.697258 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.690603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:46307 (local address 127.2.147.195:51183)
0926 07:28:39.691305 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:39.691322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:39.691434 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:39.691726 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:39.691728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:39.691745 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:39.692015 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.692021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.693136 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.693141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.695035 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:39.695045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.696308 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.696314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:39.696486 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.696519 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:28:39.696619 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:28:39.696870 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":2,"thread_start_us":328,"threads_started":1}
I20260926 07:28:39.697786 13108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a647ef71f1a84a079f8c5b4f7dcae26d" candidate_uuid: "b375bd488e7a43f9958b2e4ffb219e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" is_pre_election: true
I20260926 07:28:39.698094 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a647ef71f1a84a079f8c5b4f7dcae26d" candidate_uuid: "b375bd488e7a43f9958b2e4ffb219e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" is_pre_election: true
I20260926 07:28:39.698266 13108 raft_consensus.cc:2454] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b375bd488e7a43f9958b2e4ffb219e96 in term 0.
I20260926 07:28:39.698581 13242 raft_consensus.cc:2454] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b375bd488e7a43f9958b2e4ffb219e96 in term 0.
I20260926 07:28:39.698954 13311 leader_election.cc:302] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ede1b73d994a75b0eb5fac4327d62e, b375bd488e7a43f9958b2e4ffb219e96; no voters:
I20260926 07:28:39.699368 13447 raft_consensus.cc:2781] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:39.699522 13447 raft_consensus.cc:482] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:39.699620 13447 raft_consensus.cc:3037] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:39.701763 13447 raft_consensus.cc:504] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.702442 13447 leader_election.cc:288] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [CANDIDATE]: Term 1 election: Requested vote from peers a5ede1b73d994a75b0eb5fac4327d62e (127.2.147.193:33975), e05e00a01f7e4ca0953833dcd2c722ec (127.2.147.194:46307)
I20260926 07:28:39.702816 13108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a647ef71f1a84a079f8c5b4f7dcae26d" candidate_uuid: "b375bd488e7a43f9958b2e4ffb219e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ede1b73d994a75b0eb5fac4327d62e"
I20260926 07:28:39.703003 13108 raft_consensus.cc:3037] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:39.702925 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a647ef71f1a84a079f8c5b4f7dcae26d" candidate_uuid: "b375bd488e7a43f9958b2e4ffb219e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05e00a01f7e4ca0953833dcd2c722ec"
I20260926 07:28:39.703172 13242 raft_consensus.cc:3037] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:39.705240 13108 raft_consensus.cc:2454] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b375bd488e7a43f9958b2e4ffb219e96 in term 1.
I20260926 07:28:39.705466 13242 raft_consensus.cc:2454] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b375bd488e7a43f9958b2e4ffb219e96 in term 1.
I20260926 07:28:39.705587 13311 leader_election.cc:302] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ede1b73d994a75b0eb5fac4327d62e, b375bd488e7a43f9958b2e4ffb219e96; no voters:
I20260926 07:28:39.705855 13447 raft_consensus.cc:2781] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:39.706327 13447 raft_consensus.cc:686] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 1 LEADER]: Becoming Leader. State: Replica: b375bd488e7a43f9958b2e4ffb219e96, State: Running, Role: LEADER
I20260926 07:28:39.706871 13447 consensus_queue.cc:237] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } }
I20260926 07:28:39.711364 12964 catalog_manager.cc:5495] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 reported cstate change: term changed from 0 to 1, leader changed from <none> to b375bd488e7a43f9958b2e4ffb219e96 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "b375bd488e7a43f9958b2e4ffb219e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b375bd488e7a43f9958b2e4ffb219e96" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 46557 } health_report { overall_health: HEALTHY } } }
I20260926 07:28:39.731191 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.725795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.2.147.195:46557)
0926 07:28:39.726028 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.726034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.726053 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:28:39.726183 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.726190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.726303 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.726472 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:39.726850 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.727771 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.729085 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.729496 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.729642 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.730122 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:39.730635 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:39.730794 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.730853 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.730932 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:28:39.739573 13108 raft_consensus.cc:1261] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 1 FOLLOWER]: Refusing update from remote peer b375bd488e7a43f9958b2e4ffb219e96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:39.739571 13242 raft_consensus.cc:1261] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 1 FOLLOWER]: Refusing update from remote peer b375bd488e7a43f9958b2e4ffb219e96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:39.740574 13449 consensus_queue.cc:1035] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e05e00a01f7e4ca0953833dcd2c722ec" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:39.740985 13447 consensus_queue.cc:1035] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ede1b73d994a75b0eb5fac4327d62e" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:39.753202 13452 mvcc.cc:205] Tried to move back new op lower bound from 7333510020042338304 to 7333510019923726336. Current Snapshot: MvccSnapshot[applied={T|T < 7333510020042338304}]
I20260926 07:28:39.765368 12964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:39.765666 12964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:39.769546 12964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab077aa793d443089120e119f07c23f: Sending DeleteTablet for 3 replicas of tablet a647ef71f1a84a079f8c5b4f7dcae26d
I20260926 07:28:39.770671 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:39 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:39196
I20260926 07:28:39.770782 13355 tablet_service.cc:1514] Processing DeleteTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:39 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:42892
I20260926 07:28:39.771070 13222 tablet_service.cc:1514] Processing DeleteTablet for tablet a647ef71f1a84a079f8c5b4f7dcae26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:39 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:48186
I20260926 07:28:39.773437 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 13025
I20260926 07:28:39.799759 13464 tablet_replica.cc:330] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: stopping tablet replica
I20260926 07:28:39.802241 13464 raft_consensus.cc:2229] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:39.804041 13464 raft_consensus.cc:2258] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:39.809500 13464 ts_tablet_manager.cc:1905] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:39.816604 13463 tablet_replica.cc:330] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: stopping tablet replica
I20260926 07:28:39.817042 13463 raft_consensus.cc:2229] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:39.817641 13463 raft_consensus.cc:2258] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:39.819862 13462 tablet_replica.cc:330] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: stopping tablet replica
I20260926 07:28:39.820294 13462 raft_consensus.cc:2229] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:39.820837 13462 raft_consensus.cc:2258] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:39.821200 13463 ts_tablet_manager.cc:1905] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:39.823027 13462 ts_tablet_manager.cc:1905] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:28:39 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:28:39.829061 13463 ts_tablet_manager.cc:1918] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:39.829237 13463 log.cc:1196] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/a647ef71f1a84a079f8c5b4f7dcae26d
I20260926 07:28:39.829842 13463 ts_tablet_manager.cc:1939] T a647ef71f1a84a079f8c5b4f7dcae26d P b375bd488e7a43f9958b2e4ffb219e96: Deleting consensus metadata
I20260926 07:28:39.831561 13462 ts_tablet_manager.cc:1918] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:39.831720 13462 log.cc:1196] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/a647ef71f1a84a079f8c5b4f7dcae26d
I20260926 07:28:39.831881 12951 catalog_manager.cc:4841] TS b375bd488e7a43f9958b2e4ffb219e96 (127.2.147.195:46557): tablet a647ef71f1a84a079f8c5b4f7dcae26d (table test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]) successfully deleted
I20260926 07:28:39.832248 13462 ts_tablet_manager.cc:1939] T a647ef71f1a84a079f8c5b4f7dcae26d P a5ede1b73d994a75b0eb5fac4327d62e: Deleting consensus metadata
I20260926 07:28:39.835286 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.774869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.2.147.193:33975)
0926 07:28:39.775100 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.775109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.798622 (+ 23513us) server_negotiation.cc:407] Connection header received
0926 07:28:39.800120 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.800130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.800244 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.800398 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:39.804528 (+ 4130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.806100 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.818043 (+ 11943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.818557 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.818646 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.825582 (+ 6936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:39.825598 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:39.825616 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:39.825668 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:39.830014 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.832297 (+ 2283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.832308 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.832312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.832539 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.833148 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.833153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.833157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.833690 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:39.834003 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.834640 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.834872 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:28:39.835469 12951 catalog_manager.cc:4841] TS a5ede1b73d994a75b0eb5fac4327d62e (127.2.147.193:33975): tablet a647ef71f1a84a079f8c5b4f7dcae26d (table test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]) successfully deleted
I20260926 07:28:39.835981 13464 ts_tablet_manager.cc:1918] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:39.836162 13464 log.cc:1196] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/a647ef71f1a84a079f8c5b4f7dcae26d
I20260926 07:28:39.836782 13464 ts_tablet_manager.cc:1939] T a647ef71f1a84a079f8c5b4f7dcae26d P e05e00a01f7e4ca0953833dcd2c722ec: Deleting consensus metadata
I20260926 07:28:39.837121 13148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:39.838714 12951 catalog_manager.cc:4841] TS e05e00a01f7e4ca0953833dcd2c722ec (127.2.147.194:46307): tablet a647ef71f1a84a079f8c5b4f7dcae26d (table test-table [id=d926f9082d4643a3ac260f1d8b5f4c18]) successfully deleted
I20260926 07:28:39.928849 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 13025
I20260926 07:28:39.946842 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 13159
Sep 26 07:28:39 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:28:39.963079 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:39.947653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.2.147.194:46307)
0926 07:28:39.947869 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:39.947876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:39.947897 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:28:39.947990 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:39.947996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:39.948077 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:39.948185 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:39.949333 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.950497 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.951750 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:39.952372 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:39.952498 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:39.956434 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:39.956453 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:39.956474 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:39.956520 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:39.960312 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.960984 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.960994 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.960998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.961157 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:39.961483 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:39.961488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:39.961492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:39.961982 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:39.962273 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:39.962478 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:28:39.962705 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:28:39.967909 13282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:40.102494 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 13159
I20260926 07:28:40.125263 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 13292
Sep 26 07:28:40 dist-test-slave-1dwd krb5kdc[12919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:28:40.143092 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:40.126169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.2.147.195:46557)
0926 07:28:40.126367 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:40.126372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:40.126395 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:28:40.126514 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:40.126519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:40.126612 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:40.126726 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:40.128103 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:40.129385 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:40.130666 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:40.131393 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:40.131547 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:40.135674 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:40.135695 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:40.135720 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:40.135767 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:40.140168 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:40.140817 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:40.140827 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:40.140830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:40.141034 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:40.141355 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:40.141361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:40.141364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:40.141826 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:40.142170 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:40.142458 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:28:40.142674 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:28:40.143862 13415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:40.264851 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 13292
I20260926 07:28:40.284354 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 12932
I20260926 07:28:40.285441 12994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:28:40.392052 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 12932
2026-09-26T07:28:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4152 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:40 dist-test-slave-1dwd krb5kdc[13472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:40 dist-test-slave-1dwd krb5kdc[13472](info): set up 2 sockets
Sep 26 07:28:40 dist-test-slave-1dwd krb5kdc[13472](info): commencing operation
krb5kdc: starting...
W20260926 07:28:41.480165 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:28:41 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangeradmin_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangeradmin_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangerlookup_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangerlookup_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
I20260926 07:28:41.794155 2639 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/postgres -l logfile start
2026-09-26 07:28:44.267 UTC [13502] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:28:44.267 UTC [13502] LOG: listening on IPv4 address "127.2.147.212", port 41071
2026-09-26 07:28:44.270 UTC [13502] LOG: listening on Unix socket "/tmp/.s.PGSQL.41071"
2026-09-26 07:28:44.292 UTC [13505] LOG: database system was shut down at 2026-09-26 07:28:42 UTC
2026-09-26 07:28:44.295 UTC [13502] LOG: database system is ready to accept connections
I20260926 07:28:45.263922 2639 mini_postgres.cc:96] Postgres bound to 41071
2026-09-26 07:28:45.281 UTC [13513] FATAL: database "slave" does not exist
127.2.147.212:41071 - accepting connections
I20260926 07:28:45.281932 2639 mini_ranger.cc:161] Starting Ranger...
I20260926 07:28:45.307950 2639 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:28:45.800823 2639 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:28:45.800957 2639 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin
2026-09-26 07:28:46,093 [I] DB FLAVOR :POSTGRES
2026-09-26 07:28:46,094 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:28:46,094 [I] Checking connection..
2026-09-26 07:28:46,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:28:46,334 [I] Checking connection passed.
2026-09-26 07:28:46,335 [I] --------- Verifying version history table ---------
2026-09-26 07:28:46,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:28:46,594 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:28:46,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:28:46,843 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:28:46,844 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:28:46,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:28:47.101 UTC [13577] WARNING: there is no transaction in progress
2026-09-26 07:28:47,110 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:28:47,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:28:47,358 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:28:47,358 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:28:47,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:28:47,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:28:47,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1dwd.c.gcp-upstream.internal','N') ;"
2026-09-26 07:28:48,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:28:48,330 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:28:48,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:28:48,560 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:28:48,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:28:48,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:28:49,050 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:28:49,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:28:50.025 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.041 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.058 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.074 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.091 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.110 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.167 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.174 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.188 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.199 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.209 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.223 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.233 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.239 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.246 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50.253 UTC [13685] WARNING: there is no transaction in progress
2026-09-26 07:28:50,920 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:28:50,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:28:51,151 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:28:51,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:28:51,406 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:28:51,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:28:51,663 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:28:51,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:28:51,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:28:52,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1dwd.c.gcp-upstream.internal';"
2026-09-26 07:28:52,367 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:28:52,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:28:52,603 [I] DB_PATCHES have already been applied
I20260926 07:28:52.609671 2639 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLNb1Qs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:28:52.609797 2639 mini_ranger.cc:193] Using host: 127.2.147.212
I20260926 07:28:52.625911 2639 mini_ranger.cc:227] Ranger admin URL: http://127.2.147.212:39173
Sep 26, 2026 7:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:28:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:28:53.740986 2639 mini_ranger.cc:160] Time spent starting Ranger: real 8.459s user 0.000s sys 0.040s
Sep 26, 2026 7:29:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:29:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 7:29:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:29:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:29:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:29:15.024123 2639 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangerkms_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/rangerkms_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.147.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.147.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:29:15 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407755, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:29:15.228157 2639 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:29:15.248245 2639 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:29:16.098819 2639 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:29:16.098982 2639 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms
I20260926 07:29:16.099112 2639 mini_ranger_kms.cc:227] Using postgres at 127.2.147.212:41071
2026-09-26 07:29:16,512 [I] DB FLAVOR :POSTGRES
2026-09-26 07:29:16,512 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:29:16,513 [I] Checking connection
2026-09-26 07:29:16,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:29:16,762 [I] connection success
2026-09-26 07:29:16,762 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:29:16,762 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:29:16,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:29:17,018 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:29:17,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:29:17,018 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:29:17,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:29:17,279 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:29:17.723771 2639 mini_ranger_kms.cc:310] Created kms service
I20260926 07:29:17.878471 2639 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:29:17.921618 2639 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:29:17 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407755, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:18.315109 2639 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:29:18.315567 2639 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:29:18.315622 2639 mini_ranger_kms.cc:240] Using host: 127.2.147.212
I20260926 07:29:18.336009 2639 mini_ranger_kms.cc:276] Ranger KMS PID: 13865
I20260926 07:29:18.336208 2639 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.147.212:57093
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:29:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:29:20.695359 2639 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.467s user 0.156s sys 0.278s
I20260926 07:29:20.695605 2639 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:29:20.695811 2639 mini_ranger_kms.cc:401] 127.2.147.212:57093/kms/v1/keys
Sep 26 07:29:20 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407760, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:29:21 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407760, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:29:21 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407760, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:29:23 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:29:23.870388 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40405
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:40405
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:57093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:24.235245 13916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:24.235597 13916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:24.235771 13916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:24.243180 13916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:24.243376 13916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:24.243477 13916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:24.243571 13916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:24.243655 13916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:24.255165 13916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.147.212:57093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:40405
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:29:24.256616 13916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:24.257994 13916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:24.271416 13921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:24.271364 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:24.271680 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:24.272670 13916 server_base.cc:1034] running on GCE node
I20260926 07:29:24.273464 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:24.274988 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:24.276222 13916 hybrid_clock.cc:648] HybridClock initialized: now 1790407764276173 us; error 47 us; skew 500 ppm
I20260926 07:29:24.276587 13916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:29:24 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407764, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:24.280095 13916 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:29:24.281976 13916 webserver.cc:460] Webserver started at http://127.2.147.254:37831/ using document root <none> and password file <none>
I20260926 07:29:24.282881 13916 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:24.283018 13916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:24.283420 13916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:29:24 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407764, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:24.347817 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "4c1189db5c5543af94255e0c09607662"
format_stamp: "Formatted at 2026-09-26 07:29:24 on dist-test-slave-1dwd"
server_key: "450cd62eb40dbf1a482e9ed4dee5bbcc"
server_key_iv: "e4aa7626c3fbd12ff05528309d176041"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:24.348668 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "4c1189db5c5543af94255e0c09607662"
format_stamp: "Formatted at 2026-09-26 07:29:24 on dist-test-slave-1dwd"
server_key: "450cd62eb40dbf1a482e9ed4dee5bbcc"
server_key_iv: "e4aa7626c3fbd12ff05528309d176041"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:24.354179 13916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 07:29:24.374192 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:24.375916 13916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:24.376123 13916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "4c1189db5c5543af94255e0c09607662"
format_stamp: "Formatted at 2026-09-26 07:29:24 on dist-test-slave-1dwd"
server_key: "450cd62eb40dbf1a482e9ed4dee5bbcc"
server_key_iv: "e4aa7626c3fbd12ff05528309d176041"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:24.376367 13916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:24.398360 13916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:24.402329 13916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:24.402642 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:24.423844 13916 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:40405
I20260926 07:29:24.423877 13982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:40405 every 8 connection(s)
I20260926 07:29:24.425216 13916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:24.429039 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 13916
I20260926 07:29:24.429235 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:29:24.429461 13983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:24.429999 2639 external_mini_cluster.cc:1428] Setting key 6f26fc049e2795306204b4fef4cf91e6
I20260926 07:29:24.448259 13983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: Bootstrap starting.
I20260926 07:29:24.452303 13983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:24.453506 13983 log.cc:824] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:24.457322 13983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: No bootstrap required, opened a new log
Sep 26 07:29:24 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:29:24.467562 13983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } }
I20260926 07:29:24.468031 13983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:24.468129 13983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1189db5c5543af94255e0c09607662, State: Initialized, Role: FOLLOWER
I20260926 07:29:24.468779 13983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } }
I20260926 07:29:24.469065 13983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:24.469244 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:24.469470 13983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:24.472613 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } }
I20260926 07:29:24.472805 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:24.432629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.2.147.254:40405)
0926 07:29:24.443839 (+ 11210us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:24.443855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:24.446968 (+ 3113us) server_negotiation.cc:407] Connection header received
0926 07:29:24.449631 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:24.449661 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:24.450228 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:24.450823 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:24.453185 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:24.455099 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:24.456975 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:24.457948 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:24.458130 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:24.463199 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:24.463256 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:24.463281 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:24.463327 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:24.467706 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:24.468557 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:24.468566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:24.468575 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:24.468779 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:24.469252 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:24.469257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:24.469260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:24.469962 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:24.470268 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:24.470746 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 07:29:24.471258 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10932,"thread_start_us":10687,"threads_started":1}
I20260926 07:29:24.473210 13983 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1189db5c5543af94255e0c09607662; no voters:
I20260926 07:29:24.473899 13983 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:24.474022 13988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:24.477485 13988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1189db5c5543af94255e0c09607662, State: Running, Role: LEADER
I20260926 07:29:24.478119 13988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } }
I20260926 07:29:24.478534 13983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:24.489431 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1189db5c5543af94255e0c09607662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } } }
I20260926 07:29:24.489971 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:24.493817 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1189db5c5543af94255e0c09607662. Latest consensus state: current_term: 1 leader_uuid: "4c1189db5c5543af94255e0c09607662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1189db5c5543af94255e0c09607662" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40405 } } }
I20260926 07:29:24.494663 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:24.503324 13994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:24.510085 13994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:24.525904 13994 catalog_manager.cc:1347] Generated new cluster ID: d5388be6a0ca431b86985726c41285a9
I20260926 07:29:24.526059 13994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:24.536227 13983 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:29:24.565673 13994 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:24.567977 13994 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:24.593212 13994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: Generated new TSK 0
I20260926 07:29:24.594460 13994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:29:25.412384 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:57093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:29:25 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:29:25.881266 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:25.881678 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:25.881836 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:25.892417 14028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:25.892580 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:25.892761 14028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:29:25.907718 14028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.147.212:57093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:29:25.909444 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:25.911247 14028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:25.928488 14039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:25.928515 14038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:25.928800 14041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:25.930084 14028 server_base.cc:1034] running on GCE node
I20260926 07:29:25.930738 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:25.931850 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:25.933082 14028 hybrid_clock.cc:648] HybridClock initialized: now 1790407765933012 us; error 65 us; skew 500 ppm
I20260926 07:29:25.933452 14028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:29:25 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:25.937234 14028 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:29:25.938999 14028 webserver.cc:460] Webserver started at http://127.2.147.193:42117/ using document root <none> and password file <none>
I20260926 07:29:25.940078 14028 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:25.940640 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:25.941071 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:29:25 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:25.963302 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "29298516a95540519ea02b13c44d7cbc"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-1dwd"
server_key: "340a78358e06cdf39579a00aa819ad63"
server_key_iv: "5e65ddc6544a18892b3b75f2b478b375"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:25.964555 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "29298516a95540519ea02b13c44d7cbc"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-1dwd"
server_key: "340a78358e06cdf39579a00aa819ad63"
server_key_iv: "5e65ddc6544a18892b3b75f2b478b375"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:25.977739 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260926 07:29:26.005482 14048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.008569 14028 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.004s
I20260926 07:29:26.008903 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "29298516a95540519ea02b13c44d7cbc"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-1dwd"
server_key: "340a78358e06cdf39579a00aa819ad63"
server_key_iv: "5e65ddc6544a18892b3b75f2b478b375"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:26.009284 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:26.071686 14028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:26.077382 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:26.077737 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:26.085367 14028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:26.087474 14028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:26.087581 14028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.087762 14028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:26.087831 14028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:26.198939 14028 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:46611
I20260926 07:29:26.198968 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:46611 every 8 connection(s)
I20260926 07:29:26.200785 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:26.207850 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 14028
I20260926 07:29:26.208088 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:26.209074 2639 external_mini_cluster.cc:1428] Setting key 1e20521fa42ce7d9bf538a2082338749
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:29:26.247893 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.204378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40405 (local address 127.2.147.193:60133)
0926 07:29:26.214930 (+ 10552us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:26.214996 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:26.217750 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:26.223896 (+ 6146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:26.223912 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:26.224617 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:26.225573 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:26.225599 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.227035 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.227040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:26.229009 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:26.229024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.230641 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.230647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:26.230911 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.231741 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:26.231770 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:26.235621 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:26.242930 (+ 7309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:26.242940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:26.242961 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:26.243605 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:26.244035 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:26.244040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:26.244043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:26.244290 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:26.245008 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:26.245016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:26.245508 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 07:29:26.245830 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 07:29:26.246384 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9282,"thread_start_us":9431,"threads_started":1}
I20260926 07:29:26.249316 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.203590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60133 (local address 127.2.147.254:40405)
0926 07:29:26.222918 (+ 19328us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:26.222925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:26.222949 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:29:26.223015 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:26.223020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:26.223116 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:26.223408 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:26.225789 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.226853 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.229198 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.230236 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.230349 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.235942 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:26.235948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:26.235952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:26.235992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:26.242725 (+ 6733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.243769 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.243774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.243777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.243917 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.244421 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.244425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.244428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.244790 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:26.244926 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:26.248663 (+ 3737us) server_negotiation.cc:299] Negotiation successful
0926 07:29:26.248927 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19173,"thread_start_us":19613,"threads_started":1}
I20260926 07:29:26.251717 14161 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40405
I20260926 07:29:26.252045 14161 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:26.252930 14161 heartbeater.cc:511] Master 127.2.147.254:40405 requested a full tablet report, sending...
I20260926 07:29:26.255481 13948 ts_manager.cc:194] Registered new tserver with Master: 29298516a95540519ea02b13c44d7cbc (127.2.147.193:46611)
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:29:26.257351 13948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:60133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:29:26.298427 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:57093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
W20260926 07:29:26.780522 14167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:26.781031 14167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:26.781220 14167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:26.793097 14167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:26.793358 14167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:26.793604 14167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:29:26.809940 14167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.147.212:57093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:29:26.811892 14167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:26.813683 14167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:26.828671 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:26.830473 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:26.831944 14184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:26.832715 14167 server_base.cc:1034] running on GCE node
I20260926 07:29:26.833372 14167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:26.834419 14167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:26.835628 14167 hybrid_clock.cc:648] HybridClock initialized: now 1790407766835564 us; error 57 us; skew 500 ppm
I20260926 07:29:26.835949 14167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:26.839501 14167 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:29:26.841341 14167 webserver.cc:460] Webserver started at http://127.2.147.194:33651/ using document root <none> and password file <none>
I20260926 07:29:26.842247 14167 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:26.842394 14167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:26.842756 14167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:26.859462 14167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "7d1f3811db9d4a58916c0aab445452d5"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-1dwd"
server_key: "2f71484c023ca9a33d1695968b45abca"
server_key_iv: "9d53283599ce5c6d060b1f93eb5e3b9c"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:26.860329 14167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "7d1f3811db9d4a58916c0aab445452d5"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-1dwd"
server_key: "2f71484c023ca9a33d1695968b45abca"
server_key_iv: "9d53283599ce5c6d060b1f93eb5e3b9c"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:26.865938 14167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:29:26.882743 14191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.884413 14167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:29:26.884596 14167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "7d1f3811db9d4a58916c0aab445452d5"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-1dwd"
server_key: "2f71484c023ca9a33d1695968b45abca"
server_key_iv: "9d53283599ce5c6d060b1f93eb5e3b9c"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:26.884837 14167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:26.901808 14167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:26.906409 14167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:26.906724 14167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:26.908314 14167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:26.909744 14167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:26.909817 14167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.909952 14167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:26.910004 14167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.950955 14167 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:33853
I20260926 07:29:26.950979 14303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:33853 every 8 connection(s)
I20260926 07:29:26.952317 14167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:26.953157 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 14167
I20260926 07:29:26.953369 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:26.954059 2639 external_mini_cluster.cc:1428] Setting key 055b626628168389173cbfbca16f81e0
Sep 26 07:29:26 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
I20260926 07:29:26.988889 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.956848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:38375 (local address 127.2.147.254:40405)
0926 07:29:26.971591 (+ 14743us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:26.971596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:26.972265 (+ 669us) server_negotiation.cc:407] Connection header received
0926 07:29:26.974131 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:26.974136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:26.974248 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:26.974362 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:26.975926 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.976654 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.978260 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.978654 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.978787 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.982500 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:26.982505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:26.982508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:26.982543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:26.985127 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.985841 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.985845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.985848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.985959 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.986443 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.986449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.986453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.987346 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:26.987517 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:26.988291 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 07:29:26.988515 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14632,"thread_start_us":14533,"threads_started":1}
I20260926 07:29:26.995997 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.956238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40405 (local address 127.2.147.194:38375)
0926 07:29:26.972068 (+ 15830us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:26.972130 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:26.973855 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:26.974627 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:26.974636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:26.975077 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:26.975762 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:26.975779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.976825 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.976828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:26.978077 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:26.978087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.978882 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.978889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:26.979260 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.980070 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:26.980103 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:26.982282 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:26.985268 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:26.985278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:26.985300 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:26.985714 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:26.986106 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:26.986109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:26.986111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:26.986297 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:26.987484 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:26.987489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:26.987819 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 07:29:26.988181 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 07:29:26.988641 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15564,"thread_start_us":15379,"threads_started":1}
I20260926 07:29:26.997547 14304 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40405
I20260926 07:29:26.997880 14304 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:26.998622 14304 heartbeater.cc:511] Master 127.2.147.254:40405 requested a full tablet report, sending...
I20260926 07:29:27.000130 13948 ts_manager.cc:194] Registered new tserver with Master: 7d1f3811db9d4a58916c0aab445452d5 (127.2.147.194:33853)
I20260926 07:29:27.000944 13948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:38375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:29:27.020188 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:57093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:29:27.261099 14161 heartbeater.cc:503] Master 127.2.147.254:40405 was elected leader, sending a full tablet report...
W20260926 07:29:27.385048 14310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:27.385341 14310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:27.385432 14310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:27.393237 14310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:27.393362 14310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:27.393472 14310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:29:27.403303 14310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.147.212:57093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:40405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:29:27.404697 14310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:27.405925 14310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:27.419472 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:27.419565 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:27.419572 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:27.420410 14310 server_base.cc:1034] running on GCE node
I20260926 07:29:27.421036 14310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:27.421933 14310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:27.423148 14310 hybrid_clock.cc:648] HybridClock initialized: now 1790407767423085 us; error 60 us; skew 500 ppm
I20260926 07:29:27.423478 14310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:29:27 dist-test-slave-1dwd krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407767, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:27.426681 14310 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:29:27.431944 14310 webserver.cc:460] Webserver started at http://127.2.147.195:43435/ using document root <none> and password file <none>
I20260926 07:29:27.432655 14310 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:27.432796 14310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:27.433128 14310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:29:27 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407767, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:29:27.447472 14310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "299c955861d046008b9ce8354adfeba9"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-1dwd"
server_key: "a181f67581a4c00fce9913b6866a6d4f"
server_key_iv: "91867463011f5350e6d7bc35eb737267"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:27.448817 14310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "299c955861d046008b9ce8354adfeba9"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-1dwd"
server_key: "a181f67581a4c00fce9913b6866a6d4f"
server_key_iv: "91867463011f5350e6d7bc35eb737267"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:27.454382 14310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:29:27.471576 14325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:27.473294 14310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:29:27.473479 14310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "299c955861d046008b9ce8354adfeba9"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-1dwd"
server_key: "a181f67581a4c00fce9913b6866a6d4f"
server_key_iv: "91867463011f5350e6d7bc35eb737267"
server_key_version: "kuduclusterkey@0"
I20260926 07:29:27.473721 14310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:27.524981 14310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:27.528736 14310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:27.528977 14310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:27.529932 14310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:27.531180 14310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:27.531318 14310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:27.531410 14310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:27.531456 14310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:27.566427 14310 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:36665
I20260926 07:29:27.566474 14437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:36665 every 8 connection(s)
I20260926 07:29:27.567863 14310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:27.577142 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 14310
I20260926 07:29:27.577342 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:27.577775 2639 external_mini_cluster.cc:1428] Setting key 8babdc5fab8eea25e4b3399cac404765
Sep 26 07:29:27 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407767, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:29:27.587385 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.570352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:41135 (local address 127.2.147.254:40405)
0926 07:29:27.571250 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.571257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.571884 (+ 627us) server_negotiation.cc:407] Connection header received
0926 07:29:27.573620 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.573625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.573707 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.573832 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:27.575187 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.575939 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.577613 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.578185 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.578316 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.582087 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:27.582092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:27.582095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:27.582126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:27.584611 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:27.585326 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:27.585330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:27.585333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:27.585450 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:27.585823 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:27.585826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:27.585828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:27.586082 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:27.586195 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.586867 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.587042 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":202,"threads_started":1}
I20260926 07:29:27.588356 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.570993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:40405 (local address 127.2.147.195:41135)
0926 07:29:27.571724 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.571766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.573367 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.574053 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.574060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.574470 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:27.575015 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.575030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.576094 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.576099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.577471 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.577482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.578452 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.578463 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.578700 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.579572 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:27.579605 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:27.581873 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:27.584759 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:27.584767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:27.584782 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:27.585224 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:27.585544 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:27.585547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:27.585550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:27.585737 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:27.586187 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:27.586192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:27.586521 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.586757 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.587175 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":334,"threads_started":1}
I20260926 07:29:27.589726 14438 heartbeater.cc:347] Connected to a master server at 127.2.147.254:40405
I20260926 07:29:27.590041 14438 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:27.590735 14438 heartbeater.cc:511] Master 127.2.147.254:40405 requested a full tablet report, sending...
I20260926 07:29:27.592978 13947 ts_manager.cc:194] Registered new tserver with Master: 299c955861d046008b9ce8354adfeba9 (127.2.147.195:36665)
I20260926 07:29:27.593569 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:27.594060 13947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:41135
I20260926 07:29:27.610477 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.598145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.2.147.254:40405)
0926 07:29:27.598273 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.598277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.598695 (+ 418us) server_negotiation.cc:407] Connection header received
0926 07:29:27.598846 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.598850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.598916 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.598993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:27.600390 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.601068 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.602355 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.602732 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.602823 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.605386 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:27.605391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:27.605394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:27.605423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:27.607817 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:27.608763 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:27.608767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:27.608769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:27.608901 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:27.609214 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:27.609217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:27.609219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:27.609480 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:27.609671 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.609917 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.610119 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:29:27.615244 13947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:27.622144 13947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:27.647826 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.635298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:46611 (local address 127.0.0.1:44766)
0926 07:29:27.637162 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.637182 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.637315 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.638777 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.638785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.638817 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:27.639344 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.639367 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.641323 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.641331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.644600 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.644612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.646988 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.646994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.647175 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.647227 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.647305 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.647454 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"mutex_wait_us":59,"thread_start_us":1568,"threads_started":1}
I20260926 07:29:27.648180 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.634968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.2.147.195:36665)
0926 07:29:27.638239 (+ 3271us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.638247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.638279 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:29:27.638367 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.638372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.638527 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.638711 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:27.639757 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.641446 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.644869 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.646237 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.646372 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.646618 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.647600 (+ 982us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.647799 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3098,"thread_start_us":2978,"threads_started":1}
I20260926 07:29:27.648597 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.634748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36665 (local address 127.0.0.1:33106)
0926 07:29:27.635575 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.635636 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.635849 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.639002 (+ 3153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.639006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.639025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:27.639572 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.639583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.641673 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.641679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.644712 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.644727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.646722 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.646734 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.646975 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.647077 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.647230 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.647461 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":131,"thread_start_us":283,"threads_started":1}
I20260926 07:29:27.649212 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.636948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33853 (local address 127.0.0.1:41402)
0926 07:29:27.640488 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.640505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.640620 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.641389 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.641393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.641421 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:27.641971 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.641983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.644402 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.644408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.646025 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.646038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.648430 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.648437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.648604 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.648627 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.648744 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.648874 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3408,"thread_start_us":3282,"threads_started":1}
I20260926 07:29:27.649439 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.637568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.2.147.194:33853)
0926 07:29:27.640084 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.640093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.640727 (+ 634us) server_negotiation.cc:407] Connection header received
0926 07:29:27.640870 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.640878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.641119 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.641296 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:27.642262 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.644255 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.646203 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.647847 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.647996 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.648239 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.648881 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.649076 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2357,"thread_start_us":2243,"threads_started":1}
I20260926 07:29:27.649478 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.634963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.2.147.193:46611)
0926 07:29:27.637897 (+ 2934us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.637909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.637939 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:29:27.638049 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.638242 (+ 193us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.638507 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.639332 (+ 825us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:27.639585 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.641169 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.644759 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.646237 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.646372 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.646626 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.649000 (+ 2374us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.649169 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2758,"thread_start_us":2606,"threads_started":1}
I20260926 07:29:27.653043 14096 tablet_service.cc:1467] Processing CreateTablet for tablet e540dccb23e24e9a835e742ee85ea502 (DEFAULT_TABLE table=test-table [id=64e0bbca2fb74a208211096fea8e5032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:27.653093 14373 tablet_service.cc:1467] Processing CreateTablet for tablet e540dccb23e24e9a835e742ee85ea502 (DEFAULT_TABLE table=test-table [id=64e0bbca2fb74a208211096fea8e5032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:27.653095 14239 tablet_service.cc:1467] Processing CreateTablet for tablet e540dccb23e24e9a835e742ee85ea502 (DEFAULT_TABLE table=test-table [id=64e0bbca2fb74a208211096fea8e5032]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:27.654435 14096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540dccb23e24e9a835e742ee85ea502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:27.654534 14239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540dccb23e24e9a835e742ee85ea502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:27.654533 14373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540dccb23e24e9a835e742ee85ea502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:27.666776 14457 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Bootstrap starting.
I20260926 07:29:27.670181 14457 tablet_bootstrap.cc:654] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:27.670499 14458 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Bootstrap starting.
I20260926 07:29:27.671622 14457 log.cc:824] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:27.674748 14459 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Bootstrap starting.
I20260926 07:29:27.675344 14458 tablet_bootstrap.cc:654] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:27.675647 14457 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: No bootstrap required, opened a new log
I20260926 07:29:27.675926 14457 ts_tablet_manager.cc:1397] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 07:29:27.676625 14458 log.cc:824] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:27.678166 14459 tablet_bootstrap.cc:654] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:27.679857 14459 log.cc:824] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:27.680503 14458 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: No bootstrap required, opened a new log
I20260926 07:29:27.680751 14458 ts_tablet_manager.cc:1397] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 07:29:27.682807 14459 tablet_bootstrap.cc:492] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: No bootstrap required, opened a new log
I20260926 07:29:27.683044 14459 ts_tablet_manager.cc:1397] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 07:29:27.685323 14457 raft_consensus.cc:348] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.685904 14457 raft_consensus.cc:374] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:27.686029 14457 raft_consensus.cc:729] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29298516a95540519ea02b13c44d7cbc, State: Initialized, Role: FOLLOWER
I20260926 07:29:27.686719 14457 consensus_queue.cc:260] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.688637 14457 ts_tablet_manager.cc:1428] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 07:29:27.690716 14458 raft_consensus.cc:348] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.691056 14458 raft_consensus.cc:374] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:27.691175 14458 raft_consensus.cc:729] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1f3811db9d4a58916c0aab445452d5, State: Initialized, Role: FOLLOWER
I20260926 07:29:27.691756 14458 consensus_queue.cc:260] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.693544 14304 heartbeater.cc:503] Master 127.2.147.254:40405 was elected leader, sending a full tablet report...
I20260926 07:29:27.695446 14458 ts_tablet_manager.cc:1428] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260926 07:29:27.695533 14459 raft_consensus.cc:348] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.695968 14459 raft_consensus.cc:374] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:27.696077 14459 raft_consensus.cc:729] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299c955861d046008b9ce8354adfeba9, State: Initialized, Role: FOLLOWER
I20260926 07:29:27.696735 14459 consensus_queue.cc:260] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.697919 14438 heartbeater.cc:503] Master 127.2.147.254:40405 was elected leader, sending a full tablet report...
I20260926 07:29:27.698969 14459 ts_tablet_manager.cc:1428] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260926 07:29:27.719831 14305 tablet.cc:2369] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:29:27.719831 14162 tablet.cc:2369] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:29:27.819490 14439 tablet.cc:2369] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:27.873265 14465 raft_consensus.cc:482] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:27.873548 14465 raft_consensus.cc:504] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.875311 14465 leader_election.cc:288] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29298516a95540519ea02b13c44d7cbc (127.2.147.193:46611), 7d1f3811db9d4a58916c0aab445452d5 (127.2.147.194:33853)
I20260926 07:29:27.881583 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.875906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:55801 (local address 127.2.147.193:46611)
0926 07:29:27.876126 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.876133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.876152 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:29:27.876243 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.876248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.876326 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.876422 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:27.876854 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.877945 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.879818 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.880714 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.880865 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.880940 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.881207 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.881334 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:29:27.881588 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.875481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:46611 (local address 127.2.147.195:55801)
0926 07:29:27.875729 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.875757 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.875887 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.876457 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.876460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.876486 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:27.876714 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.876726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.878077 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.878082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.879667 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.879684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.880856 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.880862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.881033 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.881056 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.881161 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.881300 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 07:29:27.882020 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.875733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:51951 (local address 127.2.147.194:33853)
0926 07:29:27.875975 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.875983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.876569 (+ 586us) server_negotiation.cc:407] Connection header received
0926 07:29:27.876748 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.876756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.876843 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.876924 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:27.877522 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.878326 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.880206 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.881080 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.881252 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.881322 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.881636 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.881759 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:29:27.882112 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.875746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33853 (local address 127.2.147.195:51951)
0926 07:29:27.876463 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:27.876481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:27.876606 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:27.876961 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:27.876966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:27.876991 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:27.877355 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.877366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.878532 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.878538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.880058 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:27.880070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.881272 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.881278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:27.881483 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.881506 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:29:27.881613 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:29:27.881739 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":368,"threads_started":1}
I20260926 07:29:27.883297 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540dccb23e24e9a835e742ee85ea502" candidate_uuid: "299c955861d046008b9ce8354adfeba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29298516a95540519ea02b13c44d7cbc" is_pre_election: true
I20260926 07:29:27.883827 14116 raft_consensus.cc:2454] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299c955861d046008b9ce8354adfeba9 in term 0.
I20260926 07:29:27.883919 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540dccb23e24e9a835e742ee85ea502" candidate_uuid: "299c955861d046008b9ce8354adfeba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1f3811db9d4a58916c0aab445452d5" is_pre_election: true
I20260926 07:29:27.884485 14327 leader_election.cc:302] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29298516a95540519ea02b13c44d7cbc, 299c955861d046008b9ce8354adfeba9; no voters:
I20260926 07:29:27.884490 14259 raft_consensus.cc:2454] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299c955861d046008b9ce8354adfeba9 in term 0.
I20260926 07:29:27.884862 14465 raft_consensus.cc:2781] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:27.885025 14465 raft_consensus.cc:482] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:27.885141 14465 raft_consensus.cc:3037] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:27.887382 14465 raft_consensus.cc:504] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.887943 14465 leader_election.cc:288] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [CANDIDATE]: Term 1 election: Requested vote from peers 29298516a95540519ea02b13c44d7cbc (127.2.147.193:46611), 7d1f3811db9d4a58916c0aab445452d5 (127.2.147.194:33853)
I20260926 07:29:27.888396 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540dccb23e24e9a835e742ee85ea502" candidate_uuid: "299c955861d046008b9ce8354adfeba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1f3811db9d4a58916c0aab445452d5"
I20260926 07:29:27.888409 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540dccb23e24e9a835e742ee85ea502" candidate_uuid: "299c955861d046008b9ce8354adfeba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29298516a95540519ea02b13c44d7cbc"
I20260926 07:29:27.888618 14116 raft_consensus.cc:3037] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:27.888624 14259 raft_consensus.cc:3037] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:27.890817 14116 raft_consensus.cc:2454] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299c955861d046008b9ce8354adfeba9 in term 1.
I20260926 07:29:27.891033 14259 raft_consensus.cc:2454] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299c955861d046008b9ce8354adfeba9 in term 1.
I20260926 07:29:27.891351 14327 leader_election.cc:302] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29298516a95540519ea02b13c44d7cbc, 299c955861d046008b9ce8354adfeba9; no voters:
I20260926 07:29:27.891628 14465 raft_consensus.cc:2781] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:27.892091 14465 raft_consensus.cc:686] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 1 LEADER]: Becoming Leader. State: Replica: 299c955861d046008b9ce8354adfeba9, State: Running, Role: LEADER
I20260926 07:29:27.892565 14465 consensus_queue.cc:237] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } }
I20260926 07:29:27.897184 13948 catalog_manager.cc:5495] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 299c955861d046008b9ce8354adfeba9 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "299c955861d046008b9ce8354adfeba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299c955861d046008b9ce8354adfeba9" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 } health_report { overall_health: UNKNOWN } } }
I20260926 07:29:27.941524 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.936544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.2.147.195:36665)
0926 07:29:27.936719 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.936724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.936741 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:27.936894 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.936898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.936971 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.937093 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:27.937418 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.938102 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.939539 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.939960 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.940058 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.940525 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:27.941020 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:27.941172 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.941208 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.941267 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:29:27.949409 14116 raft_consensus.cc:1261] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 1 FOLLOWER]: Refusing update from remote peer 299c955861d046008b9ce8354adfeba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:27.949407 14259 raft_consensus.cc:1261] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 1 FOLLOWER]: Refusing update from remote peer 299c955861d046008b9ce8354adfeba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:27.950343 14467 consensus_queue.cc:1035] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1f3811db9d4a58916c0aab445452d5" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:27.950687 14465 consensus_queue.cc:1035] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29298516a95540519ea02b13c44d7cbc" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:27.962971 14470 mvcc.cc:205] Tried to move back new op lower bound from 7333510217511305216 to 7333510217291309056. Current Snapshot: MvccSnapshot[applied={T|T < 7333510217511305216}]
I20260926 07:29:27.966452 14471 mvcc.cc:205] Tried to move back new op lower bound from 7333510217511305216 to 7333510217291309056. Current Snapshot: MvccSnapshot[applied={T|T < 7333510217511305216}]
I20260926 07:29:27.980746 13948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:27.981040 13948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:27.985070 13948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1189db5c5543af94255e0c09607662: Sending DeleteTablet for 3 replicas of tablet e540dccb23e24e9a835e742ee85ea502
I20260926 07:29:27.986259 14096 tablet_service.cc:1514] Processing DeleteTablet for tablet e540dccb23e24e9a835e742ee85ea502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:27 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44766
I20260926 07:29:27.986382 14373 tablet_service.cc:1514] Processing DeleteTablet for tablet e540dccb23e24e9a835e742ee85ea502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:27 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:33106
I20260926 07:29:27.986377 14239 tablet_service.cc:1514] Processing DeleteTablet for tablet e540dccb23e24e9a835e742ee85ea502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:27 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:41402
I20260926 07:29:27.989084 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 14028
I20260926 07:29:27.999377 14480 tablet_replica.cc:330] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: stopping tablet replica
I20260926 07:29:28.000037 14479 tablet_replica.cc:330] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: stopping tablet replica
I20260926 07:29:28.000775 14479 raft_consensus.cc:2229] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:28.001374 14480 raft_consensus.cc:2229] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:28.002106 14479 raft_consensus.cc:2258] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:28.002905 14480 raft_consensus.cc:2258] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:28.005800 14479 ts_tablet_manager.cc:1905] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:28.005805 14480 ts_tablet_manager.cc:1905] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:28.013262 14480 ts_tablet_manager.cc:1918] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:28.013880 14479 ts_tablet_manager.cc:1918] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:28.013931 14481 tablet_replica.cc:330] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: stopping tablet replica
I20260926 07:29:28.014241 14479 log.cc:1196] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/e540dccb23e24e9a835e742ee85ea502
I20260926 07:29:28.014415 14481 raft_consensus.cc:2229] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:28.014883 14479 ts_tablet_manager.cc:1939] T e540dccb23e24e9a835e742ee85ea502 P 29298516a95540519ea02b13c44d7cbc: Deleting consensus metadata
I20260926 07:29:28.015017 14481 raft_consensus.cc:2258] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:28.016872 13933 catalog_manager.cc:4841] TS 29298516a95540519ea02b13c44d7cbc (127.2.147.193:46611): tablet e540dccb23e24e9a835e742ee85ea502 (table test-table [id=64e0bbca2fb74a208211096fea8e5032]) successfully deleted
I20260926 07:29:28.017313 14481 ts_tablet_manager.cc:1905] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:28.022393 14481 ts_tablet_manager.cc:1918] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:28.022565 14481 log.cc:1196] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/e540dccb23e24e9a835e742ee85ea502
I20260926 07:29:28.015802 14480 log.cc:1196] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/e540dccb23e24e9a835e742ee85ea502
I20260926 07:29:28.023139 14481 ts_tablet_manager.cc:1939] T e540dccb23e24e9a835e742ee85ea502 P 7d1f3811db9d4a58916c0aab445452d5: Deleting consensus metadata
I20260926 07:29:28.023178 14480 ts_tablet_manager.cc:1939] T e540dccb23e24e9a835e742ee85ea502 P 299c955861d046008b9ce8354adfeba9: Deleting consensus metadata
Sep 26 07:29:28 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:29:28.025032 13934 catalog_manager.cc:4841] TS 7d1f3811db9d4a58916c0aab445452d5 (127.2.147.194:33853): tablet e540dccb23e24e9a835e742ee85ea502 (table test-table [id=64e0bbca2fb74a208211096fea8e5032]) successfully deleted
I20260926 07:29:28.025229 13935 catalog_manager.cc:4841] TS 299c955861d046008b9ce8354adfeba9 (127.2.147.195:36665): tablet e540dccb23e24e9a835e742ee85ea502 (table test-table [id=64e0bbca2fb74a208211096fea8e5032]) successfully deleted
I20260926 07:29:28.031870 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.990006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.2.147.193:46611)
0926 07:29:27.990184 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.990190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:28.009743 (+ 19553us) server_negotiation.cc:407] Connection header received
0926 07:29:28.009999 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:28.010005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:28.010088 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:28.010203 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:28.013782 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.014979 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.017960 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.018568 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.018708 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:28.024551 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:28.024568 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:28.024588 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:28.024631 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:28.028594 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.029398 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.029408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.029412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.029570 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.030023 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.030029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.030032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.030495 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:28.030811 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:28.031119 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:29:28.031442 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:29:28.032616 14156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:29:28.123968 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 14028
I20260926 07:29:28.147138 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 14167
Sep 26 07:29:28 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:29:28.164664 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:28.147980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.2.147.194:33853)
0926 07:29:28.148150 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:28.148154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:28.148173 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:29:28.148383 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:28.148387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:28.148459 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:28.148540 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:28.150191 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.150922 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.152792 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.153172 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.153272 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:28.158366 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:28.158382 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:28.158399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:28.158440 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:28.161744 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.162488 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.162497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.162499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.162633 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.162990 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.162996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.162999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.163454 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:28.163737 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:28.164075 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 07:29:28.164303 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:29:28.165364 14299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:29:28.254576 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 14167
I20260926 07:29:28.276678 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 14310
Sep 26 07:29:28 dist-test-slave-1dwd krb5kdc[13472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:29:28.295724 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:28.277613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.2.147.195:36665)
0926 07:29:28.277825 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:28.277832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:28.277854 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:29:28.277929 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:28.277935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:28.278018 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:28.278122 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:28.279897 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.281167 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.282609 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:28.283381 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:28.283541 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:28.288038 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:28.288064 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:28.288091 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:28.288169 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:28.292332 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.293146 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.293157 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.293161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.293393 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:28.293777 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:28.293782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:28.293785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:28.294267 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:28.294624 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:28.294795 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:29:28.295046 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:29:28.296654 14433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:29:28.419191 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 14310
I20260926 07:29:28.438867 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 13916
I20260926 07:29:28.440054 13978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:29:28.551787 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 13916
I20260926 07:29:28.574663 2639 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:29:28.910398 2639 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:29:28.910671 2639 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:29:29.256209 2639 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:29:29.256 UTC [13502] LOG: received smart shutdown request
2026-09-26 07:29:29.259 UTC [13502] LOG: background worker "logical replication launcher" (PID 13511) exited with exit code 1
2026-09-26 07:29:29.260 UTC [13506] LOG: shutting down
2026-09-26 07:29:29.423 UTC [13502] LOG: database system is shut down
2026-09-26T07:29:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49154 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:29 dist-test-slave-1dwd krb5kdc[14495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:29 dist-test-slave-1dwd krb5kdc[14495](info): set up 2 sockets
Sep 26 07:29:29 dist-test-slave-1dwd krb5kdc[14495](info): commencing operation
krb5kdc: starting...
W20260926 07:29:30.655576 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:29:30 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangeradmin_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangeradmin_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangerlookup_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangerlookup_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
I20260926 07:29:30.987866 2639 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/postgres -l logfile start
2026-09-26 07:29:33.426 UTC [14525] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:29:33.426 UTC [14525] LOG: listening on IPv4 address "127.2.147.212", port 42013
2026-09-26 07:29:33.430 UTC [14525] LOG: listening on Unix socket "/tmp/.s.PGSQL.42013"
2026-09-26 07:29:33.453 UTC [14528] LOG: database system was shut down at 2026-09-26 07:29:32 UTC
2026-09-26 07:29:33.456 UTC [14525] LOG: database system is ready to accept connections
I20260926 07:29:34.443317 2639 mini_postgres.cc:96] Postgres bound to 42013
2026-09-26 07:29:34.464 UTC [14536] FATAL: database "slave" does not exist
127.2.147.212:42013 - accepting connections
I20260926 07:29:34.465530 2639 mini_ranger.cc:161] Starting Ranger...
I20260926 07:29:34.491067 2639 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:29:34.988602 2639 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:29:34.988727 2639 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin
2026-09-26 07:29:35,274 [I] DB FLAVOR :POSTGRES
2026-09-26 07:29:35,274 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:29:35,274 [I] Checking connection..
2026-09-26 07:29:35,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:29:35,536 [I] Checking connection passed.
2026-09-26 07:29:35,536 [I] --------- Verifying version history table ---------
2026-09-26 07:29:35,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:29:35,770 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:29:35,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:29:36,029 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:29:36,029 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:29:36,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:29:36.281 UTC [14600] WARNING: there is no transaction in progress
2026-09-26 07:29:36,287 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:29:36,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:29:36,543 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:29:36,544 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:29:36,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:29:36,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:29:37,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1dwd.c.gcp-upstream.internal','N') ;"
2026-09-26 07:29:37,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:29:37,524 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:29:37,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:29:37,770 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:29:37,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:29:38,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:29:38,279 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:29:38,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:29:39.241 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.256 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.272 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.286 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.302 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.318 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.369 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.375 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.389 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.399 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.409 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.423 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.432 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.438 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.444 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:39.451 UTC [14708] WARNING: there is no transaction in progress
2026-09-26 07:29:40,081 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:29:40,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:29:40,319 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:29:40,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:29:40,566 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:29:40,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:29:40,833 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:29:40,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:29:41,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:29:41,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1dwd.c.gcp-upstream.internal';"
2026-09-26 07:29:41,595 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:29:41,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:29:41,846 [I] DB_PATCHES have already been applied
I20260926 07:29:41.852126 2639 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLNb1Qs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:29:41.852214 2639 mini_ranger.cc:193] Using host: 127.2.147.212
I20260926 07:29:41.866685 2639 mini_ranger.cc:227] Ranger admin URL: http://127.2.147.212:37719
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:29:43.016769 2639 mini_ranger.cc:160] Time spent starting Ranger: real 8.551s user 0.011s sys 0.026s
Sep 26, 2026 7:30:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:30:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 7:30:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:30:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:30:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:30:04.617871 2639 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangerkms_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/rangerkms_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.147.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.147.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.147.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/HTTP_127.2.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:30:04 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:30:04.785895 2639 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:30:04.810657 2639 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:30:05.687587 2639 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:30:05.687701 2639 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms
I20260926 07:30:05.687748 2639 mini_ranger_kms.cc:227] Using postgres at 127.2.147.212:42013
2026-09-26 07:30:06,049 [I] DB FLAVOR :POSTGRES
2026-09-26 07:30:06,049 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:30:06,049 [I] Checking connection
2026-09-26 07:30:06,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:30:06,323 [I] connection success
2026-09-26 07:30:06,323 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:30:06,323 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:30:06,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:30:06,603 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:30:06,604 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:30:06,604 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:30:06,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.147.212:42013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:30:06,897 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:30:07.378873 2639 mini_ranger_kms.cc:310] Created kms service
I20260926 07:30:07.565526 2639 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:30:07.611634 2639 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:30:07 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:08.017964 2639 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:30:08.018285 2639 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLNb1Qs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLNb1Qs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:30:08.018354 2639 mini_ranger_kms.cc:240] Using host: 127.2.147.212
I20260926 07:30:08.036964 2639 mini_ranger_kms.cc:276] Ranger KMS PID: 14888
I20260926 07:30:08.037104 2639 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.147.212:35209
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:30:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:30:10.297982 2639 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.512s user 0.045s sys 0.328s
I20260926 07:30:10.298231 2639 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:30:10.298415 2639 mini_ranger_kms.cc:401] 127.2.147.212:35209/kms/v1/keys
Sep 26 07:30:10 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:30:10 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:30:11 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.147.212@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:30:13 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:30:13.444226 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37017
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:37017
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:13.778661 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:13.778942 14939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:13.779033 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:13.786389 14939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:13.786491 14939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:13.786538 14939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:13.786617 14939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:13.786680 14939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:13.797051 14939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.147.212:35209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:37017
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:13.798295 14939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:13.799714 14939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:13.812096 14944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:13.812146 14947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:13.812167 14945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:13.812493 14939 server_base.cc:1034] running on GCE node
I20260926 07:30:13.813570 14939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:13.815022 14939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:13.816264 14939 hybrid_clock.cc:648] HybridClock initialized: now 1790407813816220 us; error 45 us; skew 500 ppm
I20260926 07:30:13.816636 14939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:13 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:13.823906 14939 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:13.825449 14939 webserver.cc:460] Webserver started at http://127.2.147.254:42363/ using document root <none> and password file <none>
I20260926 07:30:13.826197 14939 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:13.826332 14939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:13.826642 14939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:30:13 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:13.877779 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "c52cf71f51244ca499300b41175af085"
format_stamp: "Formatted at 2026-09-26 07:30:13 on dist-test-slave-1dwd"
server_key: "d557d24ecdf04bdc30aa3a96eff668bb"
server_key_iv: "a40cc28d414b6af59cda12c41d38e316"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:13.879037 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "c52cf71f51244ca499300b41175af085"
format_stamp: "Formatted at 2026-09-26 07:30:13 on dist-test-slave-1dwd"
server_key: "d557d24ecdf04bdc30aa3a96eff668bb"
server_key_iv: "a40cc28d414b6af59cda12c41d38e316"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:13.887079 14939 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 07:30:13.908881 14954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:13.910871 14939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:13.911144 14939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "c52cf71f51244ca499300b41175af085"
format_stamp: "Formatted at 2026-09-26 07:30:13 on dist-test-slave-1dwd"
server_key: "d557d24ecdf04bdc30aa3a96eff668bb"
server_key_iv: "a40cc28d414b6af59cda12c41d38e316"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:13.911461 14939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:13.934237 14939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:13.939589 14939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:13.939954 14939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:13.960852 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:37017 every 8 connection(s)
I20260926 07:30:13.960891 14939 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:37017
I20260926 07:30:13.962802 14939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:13.966861 15006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:13.968734 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 14939
I20260926 07:30:13.968945 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:30:13.969755 2639 external_mini_cluster.cc:1428] Setting key ff7df864e7da61f61a8010bcc5dc4291
I20260926 07:30:13.984460 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Bootstrap starting.
I20260926 07:30:13.989217 15006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:13.990590 15006 log.cc:824] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:13.993922 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: No bootstrap required, opened a new log
Sep 26 07:30:13 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:14.002424 15006 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:14.002854 15006 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:14.002971 15006 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52cf71f51244ca499300b41175af085, State: Initialized, Role: FOLLOWER
I20260926 07:30:14.003712 15006 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:14.003991 15006 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:14.004166 15006 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:14.004366 15006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:14.008049 15006 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:14.008677 15006 leader_election.cc:302] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52cf71f51244ca499300b41175af085; no voters:
I20260926 07:30:14.009387 15006 leader_election.cc:288] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:14.009503 15011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:14.010974 15011 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 LEADER]: Becoming Leader. State: Replica: c52cf71f51244ca499300b41175af085, State: Running, Role: LEADER
I20260926 07:30:14.011730 15006 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:14.011682 15011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:14.013752 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:13.971945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.2.147.254:37017)
0926 07:30:13.986634 (+ 14689us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:13.986654 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:13.986720 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:30:13.989255 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:13.989289 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:13.989873 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:13.990340 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:13.992177 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:13.994085 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:13.996108 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:13.997253 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:13.997441 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:14.002899 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:14.002911 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:14.002932 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:14.002988 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:14.007675 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.008492 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.008501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.008510 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.008732 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.009160 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.009168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.009174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.010109 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:14.011428 (+ 1319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:14.011771 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 07:30:14.012249 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14372,"thread_start_us":15238,"threads_started":1}
I20260926 07:30:14.014571 15012 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52cf71f51244ca499300b41175af085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } } }
I20260926 07:30:14.014955 15012 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:14.016155 15013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52cf71f51244ca499300b41175af085. Latest consensus state: current_term: 1 leader_uuid: "c52cf71f51244ca499300b41175af085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } } }
I20260926 07:30:14.016494 15013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:14.017930 15017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:14.024220 15017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:14.025110 15006 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:30:14.045028 15017 catalog_manager.cc:1347] Generated new cluster ID: 990f5ac4f0544d1cab34405e6627086b
I20260926 07:30:14.045182 15017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:14.057145 15017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:14.059159 15017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:14.073304 15017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Generated new TSK 0
I20260926 07:30:14.074563 15017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:30:14.855368 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:15 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:30:15.276789 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:15.277123 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:15.277307 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:15.286300 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:15.286517 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:15.286749 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:15.298259 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:15.300051 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:15.301849 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:15.326582 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:15.327943 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:15.334196 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:15.334192 15051 server_base.cc:1034] running on GCE node
I20260926 07:30:15.335191 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:15.336530 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:15.337785 15051 hybrid_clock.cc:648] HybridClock initialized: now 1790407815337729 us; error 48 us; skew 500 ppm
I20260926 07:30:15.338207 15051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:15 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.341792 15051 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:15.347674 15051 webserver.cc:460] Webserver started at http://127.2.147.193:39011/ using document root <none> and password file <none>
I20260926 07:30:15.348527 15051 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:15.348685 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:15.349154 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:30:15 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:15.367620 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "c0cd620420b247b29c3dbebe40ce2740"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-1dwd"
server_key: "ddc62430df63e7f95cff5922b66eef16"
server_key_iv: "2d1e41379089bd74d8f4ae519af751a3"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:15.368885 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "c0cd620420b247b29c3dbebe40ce2740"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-1dwd"
server_key: "ddc62430df63e7f95cff5922b66eef16"
server_key_iv: "2d1e41379089bd74d8f4ae519af751a3"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:15.390491 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.012s sys 0.008s
I20260926 07:30:15.416777 15071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.418918 15051 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:15.419104 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "c0cd620420b247b29c3dbebe40ce2740"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-1dwd"
server_key: "ddc62430df63e7f95cff5922b66eef16"
server_key_iv: "2d1e41379089bd74d8f4ae519af751a3"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:15.419399 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:15.436883 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:15.444006 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:15.444446 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:15.451960 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:15.460147 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:15.460280 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.460423 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:15.460507 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:30:15 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:16.127565 15051 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:35433
I20260926 07:30:16.129323 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:16.127601 15184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:35433 every 8 connection(s)
I20260926 07:30:16.136480 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15051
I20260926 07:30:16.136694 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:16.138263 2639 external_mini_cluster.cc:1428] Setting key f7ec0e1af549cdd376d573089c44c53c
Sep 26 07:30:16 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:30:16 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:30:16.186350 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:16.154462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.193:58763)
0926 07:30:16.157730 (+ 3268us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:16.157779 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:16.161379 (+ 3600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:16.162520 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:16.162547 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:16.163229 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:16.164120 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:16.164148 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.165779 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.165787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:16.168047 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:16.168067 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.169803 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.169811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:16.170122 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:16.170959 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:16.170997 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:16.174106 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:16.179099 (+ 4993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:16.179113 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:16.179133 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:16.180381 (+ 1248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:16.181275 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:16.181280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:16.181283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:16.181932 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:16.183300 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:16.183308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:16.183834 (+ 526us) client_negotiation.cc:769] Sending connection context
0926 07:30:16.184172 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 07:30:16.184795 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2922,"thread_start_us":2855,"threads_started":1}
I20260926 07:30:16.187574 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:16.155107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:58763 (local address 127.2.147.254:37017)
0926 07:30:16.157310 (+ 2203us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:16.157320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:16.160647 (+ 3327us) server_negotiation.cc:407] Connection header received
0926 07:30:16.161844 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:16.161853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:16.161956 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:16.162064 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:16.164326 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.165561 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.168228 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.169283 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.169402 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:16.174422 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:16.174431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:16.174436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:16.174485 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:16.178576 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.180738 (+ 2162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.180744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.180747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.181104 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.182093 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.182098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.182102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.182834 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:16.183833 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:16.186917 (+ 3084us) server_negotiation.cc:299] Negotiation successful
0926 07:30:16.187159 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2040,"thread_start_us":2263,"threads_started":1}
I20260926 07:30:16.189373 15185 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:16.189685 15185 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:16.190409 15185 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:16.192936 14971 ts_manager.cc:194] Registered new tserver with Master: c0cd620420b247b29c3dbebe40ce2740 (127.2.147.193:35433)
I20260926 07:30:16.195024 14971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:58763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:30:16.214362 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:16.602377 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:16.602660 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:16.602824 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:16.610142 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:16.610244 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:16.610420 15192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:16.620643 15192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:16.621937 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:16.623278 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:16.636665 15205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:16.636644 15207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:16.636722 15204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:16.637480 15192 server_base.cc:1034] running on GCE node
I20260926 07:30:16.638048 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:16.639050 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:16.640293 15192 hybrid_clock.cc:648] HybridClock initialized: now 1790407816640229 us; error 60 us; skew 500 ppm
I20260926 07:30:16.640628 15192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:16 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:16.644088 15192 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:16.645646 15192 webserver.cc:460] Webserver started at http://127.2.147.194:42789/ using document root <none> and password file <none>
I20260926 07:30:16.646365 15192 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:16.646471 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:16.646723 15192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:30:16 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:16.661861 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "fe26fee40db34d7ba6d80ae8c879adbe"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-1dwd"
server_key: "8e7b9cebbc906e4216d76aac657f4a53"
server_key_iv: "72547969e458f33b0e73003bb08c4b60"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:16.663023 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "fe26fee40db34d7ba6d80ae8c879adbe"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-1dwd"
server_key: "8e7b9cebbc906e4216d76aac657f4a53"
server_key_iv: "72547969e458f33b0e73003bb08c4b60"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:16.669864 15192 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 07:30:16.685012 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:16.686717 15192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:16.686959 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "fe26fee40db34d7ba6d80ae8c879adbe"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-1dwd"
server_key: "8e7b9cebbc906e4216d76aac657f4a53"
server_key_iv: "72547969e458f33b0e73003bb08c4b60"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:16.687261 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:16.713392 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:16.717245 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:16.717600 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:16.718732 15192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:16.720171 15192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:16.720274 15192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:16.720420 15192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:16.720492 15192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:16.752880 15192 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:45113
I20260926 07:30:16.752888 15326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:45113 every 8 connection(s)
I20260926 07:30:16.754220 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:16.763106 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15192
I20260926 07:30:16.763301 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:16.763747 2639 external_mini_cluster.cc:1428] Setting key a451b6c196ba44683cfd40864f556079
Sep 26 07:30:16 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:16.775609 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:16.756841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58221 (local address 127.2.147.254:37017)
0926 07:30:16.757298 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:16.757303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:16.758319 (+ 1016us) server_negotiation.cc:407] Connection header received
0926 07:30:16.760266 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:16.760272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:16.760365 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:16.760486 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:16.761891 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.762641 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.764422 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.764983 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.765105 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:16.768690 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:16.768699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:16.768706 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:16.768755 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:16.772430 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.773149 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.773157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.773163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.773317 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.773827 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.773830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.773832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.774122 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:16.774229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:16.775116 (+ 887us) server_negotiation.cc:299] Negotiation successful
0926 07:30:16.775352 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260926 07:30:16.777055 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:16.757472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.194:58221)
0926 07:30:16.758155 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:16.758192 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:16.759976 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:16.760728 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:16.760737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:16.761149 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:16.761720 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:16.761737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.762817 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.762821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:16.764250 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:16.764266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.765249 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.765257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:16.765584 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:16.766380 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:16.766406 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:16.768457 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:16.772582 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:16.772589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:16.772602 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:16.773007 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:16.773429 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:16.773436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:16.773441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:16.773715 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:16.774255 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:16.774266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:16.774704 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 07:30:16.775003 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 07:30:16.775591 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260926 07:30:16.778476 15327 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:16.778784 15327 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:16.779525 15327 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:16.780982 14971 ts_manager.cc:194] Registered new tserver with Master: fe26fee40db34d7ba6d80ae8c879adbe (127.2.147.194:45113)
I20260926 07:30:16.781774 14971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:58221
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:30:16.815325 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:17.145157 15333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:17.145452 15333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:17.145545 15333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:17.152805 15333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:17.152920 15333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:17.153033 15333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:17.162817 15333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:17.164112 15333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:17.165345 15333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:17.178478 15339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.178550 15341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.178553 15338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.179481 15333 server_base.cc:1034] running on GCE node
I20260926 07:30:17.180153 15333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:17.181150 15333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:17.182375 15333 hybrid_clock.cc:648] HybridClock initialized: now 1790407817182326 us; error 76 us; skew 500 ppm
I20260926 07:30:17.182725 15333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:17.185830 15333 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:17.187194 15333 webserver.cc:460] Webserver started at http://127.2.147.195:36673/ using document root <none> and password file <none>
I20260926 07:30:17.187889 15333 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:17.188051 15333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:17.188305 15333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:17.198405 15185 heartbeater.cc:503] Master 127.2.147.254:37017 was elected leader, sending a full tablet report...
I20260926 07:30:17.201373 15333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "30256e28286a409fb87e05fb0ddf1ade"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-1dwd"
server_key: "e51013cc631a0dbded85b3e70407061d"
server_key_iv: "a4a96b9ba3325489adfb40c0c1ab203a"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:17.202189 15333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "30256e28286a409fb87e05fb0ddf1ade"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-1dwd"
server_key: "e51013cc631a0dbded85b3e70407061d"
server_key_iv: "a4a96b9ba3325489adfb40c0c1ab203a"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:17.207463 15333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 07:30:17.223780 15348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.225585 15333 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:30:17.225757 15333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "30256e28286a409fb87e05fb0ddf1ade"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-1dwd"
server_key: "e51013cc631a0dbded85b3e70407061d"
server_key_iv: "a4a96b9ba3325489adfb40c0c1ab203a"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:17.225981 15333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:17.258482 15333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:17.262207 15333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:17.262503 15333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:17.263520 15333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:17.264822 15333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:17.264892 15333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.265056 15333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:17.265107 15333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.297699 15333 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:33649
I20260926 07:30:17.297744 15460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:33649 every 8 connection(s)
I20260926 07:30:17.299046 15333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:17.308446 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15333
I20260926 07:30:17.308622 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:17.309069 2639 external_mini_cluster.cc:1428] Setting key cf3a39e649302797c7af99cd2e2d2c37
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:17.319514 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.301603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:49639 (local address 127.2.147.254:37017)
0926 07:30:17.302079 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.302086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.303458 (+ 1372us) server_negotiation.cc:407] Connection header received
0926 07:30:17.305262 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.305267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.305386 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.305510 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.306967 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.307795 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.309565 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.310241 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.310373 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.313880 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.313884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.313887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.313919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.316502 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.317238 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.317242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.317245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.317429 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.317858 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.317861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.317864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.318223 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.318333 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.318954 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.319260 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260926 07:30:17.320493 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.302225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.195:49639)
0926 07:30:17.303234 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:17.303298 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:17.304967 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:17.305765 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:17.305775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:17.306197 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:17.306790 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.306807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.307950 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.307956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.309418 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.309430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.310530 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.310536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.310747 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.311568 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:17.311590 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:17.313682 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:17.316624 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.316630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.316643 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.317114 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.317542 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.317545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.317547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.317735 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.318330 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:17.318336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:17.318643 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 07:30:17.318876 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 07:30:17.319350 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":409,"threads_started":1}
I20260926 07:30:17.321769 15461 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:17.322079 15461 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:17.322870 15461 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:17.324482 14971 ts_manager.cc:194] Registered new tserver with Master: 30256e28286a409fb87e05fb0ddf1ade (127.2.147.195:33649)
I20260926 07:30:17.325304 14971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:49639
I20260926 07:30:17.336723 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:17.338307 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15051
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:30:17.354713 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.339082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.2.147.193:35433)
0926 07:30:17.339704 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.339711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.339739 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:30:17.339795 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.339801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.340006 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.340128 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.341505 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.342531 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.344169 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.344738 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.344850 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.349103 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.349116 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.349132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.349171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.352210 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.352878 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.352885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.352887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.353018 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.353339 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.353342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.353344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.353674 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.353910 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.354230 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.354434 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
I20260926 07:30:17.356097 15179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:17.441185 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15051
I20260926 07:30:17.457320 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15192
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:30:17.473346 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.458174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.2.147.194:45113)
0926 07:30:17.458733 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.458739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.458763 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:30:17.458823 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.458829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.459047 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.459174 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.460380 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.461389 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.462657 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.463189 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.463352 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.467702 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.467713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.467728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.467761 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.470722 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.471594 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.471601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.471603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.471724 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.472060 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.472063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.472065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.472384 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.472610 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.472871 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.473059 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":306,"threads_started":1}
I20260926 07:30:17.474655 15322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:17.556351 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15192
I20260926 07:30:17.572016 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15333
Sep 26 07:30:17 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:30:17.587749 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.573076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.2.147.195:33649)
0926 07:30:17.573732 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.573739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.573768 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:30:17.573826 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.573832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.574035 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.574166 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.575460 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.576450 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.577678 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.578210 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.578306 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.582474 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.582485 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.582500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.582531 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.585376 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.585988 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.585994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.585996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.586114 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.586414 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.586417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.586419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.586759 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.586986 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.587160 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.587411 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
I20260926 07:30:17.588985 15456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:17.671589 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15333
I20260926 07:30:17.687139 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 14939
I20260926 07:30:17.688478 15001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:17.795933 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 14939
I20260926 07:30:17.820056 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37017
--webserver_interface=127.2.147.254
--webserver_port=42363
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:37017
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:18.146539 15473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:18.146895 15473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:18.147049 15473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:18.154482 15473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:18.154636 15473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:18.154742 15473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:18.154841 15473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:18.154932 15473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:18.164963 15473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:37017
--ranger_config_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:18.166339 15473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:18.167698 15473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.179244 15481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.179176 15479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.179252 15478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.180032 15473 server_base.cc:1034] running on GCE node
I20260926 07:30:18.180859 15473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.182263 15473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.183483 15473 hybrid_clock.cc:648] HybridClock initialized: now 1790407818183446 us; error 52 us; skew 500 ppm
I20260926 07:30:18.183835 15473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:18 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.186996 15473 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:18.188525 15473 webserver.cc:460] Webserver started at http://127.2.147.254:42363/ using document root <none> and password file <none>
I20260926 07:30:18.189148 15473 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.189253 15473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:30:18 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:18.242594 15473 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:18.246007 15489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.247610 15473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:30:18.247815 15473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "c52cf71f51244ca499300b41175af085"
format_stamp: "Formatted at 2026-09-26 07:30:13 on dist-test-slave-1dwd"
server_key: "d557d24ecdf04bdc30aa3a96eff668bb"
server_key_iv: "a40cc28d414b6af59cda12c41d38e316"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:18.248731 15473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.276475 15473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.280761 15473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:18.281059 15473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.299649 15473 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:37017
I20260926 07:30:18.299660 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:37017 every 8 connection(s)
I20260926 07:30:18.301100 15473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:18.303802 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15473
I20260926 07:30:18.304656 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:35433
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:37017
--webserver_port=39011
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:18.307996 15541 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:18.311110 15541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Bootstrap starting.
I20260926 07:30:18.329181 15541 log.cc:824] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:18.337814 15541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:18.338341 15541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Bootstrap complete.
I20260926 07:30:18.347486 15541 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:18.348408 15541 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52cf71f51244ca499300b41175af085, State: Initialized, Role: FOLLOWER
I20260926 07:30:18.349072 15541 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:18.349326 15541 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:18.349463 15541 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:18.349645 15541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:30:18.353761 15541 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:18.354290 15541 leader_election.cc:302] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52cf71f51244ca499300b41175af085; no voters:
I20260926 07:30:18.354938 15541 leader_election.cc:288] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:30:18.355077 15545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:30:18.357398 15541 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:18.362004 15545 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [term 2 LEADER]: Becoming Leader. State: Replica: c52cf71f51244ca499300b41175af085, State: Running, Role: LEADER
I20260926 07:30:18.362610 15545 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } }
I20260926 07:30:18.365620 15546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c52cf71f51244ca499300b41175af085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } } }
I20260926 07:30:18.366060 15546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.366008 15547 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52cf71f51244ca499300b41175af085. Latest consensus state: current_term: 2 leader_uuid: "c52cf71f51244ca499300b41175af085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52cf71f51244ca499300b41175af085" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37017 } } }
I20260926 07:30:18.366321 15547 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.369184 15558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:18.374302 15558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:18.375186 15541 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:30:18.375811 15558 catalog_manager.cc:1259] Loaded cluster ID: 990f5ac4f0544d1cab34405e6627086b
I20260926 07:30:18.375965 15558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:18.379848 15558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:18.381384 15558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Loaded TSK: 0
I20260926 07:30:18.382967 15558 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:30:18.709511 15543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:18.709996 15543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:18.710253 15543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:18.721225 15543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:18.721367 15543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:18.721561 15543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:18.735798 15543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:35433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=39011
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:18.737425 15543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:18.739009 15543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.787739 15579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.790870 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.791738 15582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.793928 15543 server_base.cc:1034] running on GCE node
I20260926 07:30:18.794900 15543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.799453 15543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.800757 15543 hybrid_clock.cc:648] HybridClock initialized: now 1790407818800685 us; error 73 us; skew 500 ppm
I20260926 07:30:18.801131 15543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:18 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.804949 15543 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:18.808718 15543 webserver.cc:460] Webserver started at http://127.2.147.193:39011/ using document root <none> and password file <none>
I20260926 07:30:18.809479 15543 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.809645 15543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:30:18 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:18.836495 15543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:18.846129 15589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.848205 15543 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260926 07:30:18.848472 15543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "c0cd620420b247b29c3dbebe40ce2740"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-1dwd"
server_key: "ddc62430df63e7f95cff5922b66eef16"
server_key_iv: "2d1e41379089bd74d8f4ae519af751a3"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:18.850077 15543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.897686 15543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.903741 15543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:18.904145 15543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.919469 15543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:18.921350 15543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:18.921564 15543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 07:30:18.921754 15543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:18.921850 15543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.986157 15543 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:35433
I20260926 07:30:18.986245 15702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:35433 every 8 connection(s)
I20260926 07:30:18.987880 15543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:18.997453 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15543
I20260926 07:30:18.998503 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:45113
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:37017
--webserver_port=42789
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:19 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:19.037690 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.992127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:53407 (local address 127.2.147.254:37017)
0926 07:30:19.007693 (+ 15566us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.007720 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.011291 (+ 3571us) server_negotiation.cc:407] Connection header received
0926 07:30:19.012982 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.013008 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.013534 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.013945 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.016650 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.018807 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.021629 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.022796 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.023029 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.027653 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.027663 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.027684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.027733 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.031805 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.032803 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.032815 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.032828 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.033048 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.033695 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.033700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.033703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.034359 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.034564 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.035948 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.036358 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15181,"thread_start_us":15489,"threads_started":1}
I20260926 07:30:19.037704 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.992248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.193:53407)
0926 07:30:19.008691 (+ 16443us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.008752 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.012216 (+ 3464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.014760 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.014773 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.015479 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:19.016379 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.016408 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.019059 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.019067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.021433 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.021454 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.023108 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.023117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.023471 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.024243 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:19.024289 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:19.027307 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:19.031966 (+ 4659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.031992 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.032010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.032611 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.033220 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.033227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.033232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.033527 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.034527 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:19.034537 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:19.035370 (+ 833us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.035710 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.036306 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16096,"thread_start_us":15890,"threads_started":1}
I20260926 07:30:19.041496 15703 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:19.041814 15703 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.042634 15703 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
W20260926 07:30:19.046006 15703 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:37017 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:30:19.046232 15703 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.046494 15703 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:19.048066 15703 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.048408 15703 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
W20260926 07:30:19.049878 15703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:30:19.413074 15707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:19.413412 15707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:19.413585 15707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:19.421212 15707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:19.421367 15707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:19.421591 15707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:19.431810 15707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:45113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=42789
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:19.433152 15707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:19.434515 15707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:19.450238 15720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:19.451746 15719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:19.452683 15722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:19.453807 15707 server_base.cc:1034] running on GCE node
I20260926 07:30:19.454726 15707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:19.456058 15707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:19.457271 15707 hybrid_clock.cc:648] HybridClock initialized: now 1790407819457212 us; error 53 us; skew 500 ppm
I20260926 07:30:19.457600 15707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:19 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:19.461514 15707 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:19.464474 15707 webserver.cc:460] Webserver started at http://127.2.147.194:42789/ using document root <none> and password file <none>
I20260926 07:30:19.465212 15707 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:19.465372 15707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:30:19 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:19.488761 15707 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:30:19.492650 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:19.494521 15707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:19.494776 15707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "fe26fee40db34d7ba6d80ae8c879adbe"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-1dwd"
server_key: "8e7b9cebbc906e4216d76aac657f4a53"
server_key_iv: "72547969e458f33b0e73003bb08c4b60"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:19.496361 15707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:19.517316 15707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:19.522836 15707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:19.523267 15707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:19.531844 15707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:19.541865 15707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:19.542092 15707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 07:30:19.542275 15707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:19.542390 15707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:19.609071 15707 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:45113
I20260926 07:30:19.609149 15846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:45113 every 8 connection(s)
I20260926 07:30:19.610998 15707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:19.615453 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15707
I20260926 07:30:19.616214 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.147.212:35209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:33649
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:37017
--webserver_port=36673
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:19 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:30:19 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:19.646696 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.619438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:56521 (local address 127.2.147.254:37017)
0926 07:30:19.619972 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.619981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.625787 (+ 5806us) server_negotiation.cc:407] Connection header received
0926 07:30:19.627186 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.627193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.627309 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.627399 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.629745 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.630949 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.633762 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.634443 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.634551 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.638963 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.638971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.638976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.639019 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.642708 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.643698 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.643704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.643708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.643860 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.644449 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.644455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.644459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.644859 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.645009 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.646058 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.646321 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":340,"threads_started":1}
I20260926 07:30:19.649643 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.614521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.194:56521)
0926 07:30:19.623909 (+ 9388us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.623971 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.626798 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.628018 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.628043 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.628639 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:19.629517 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.629545 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.631222 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.631229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.633558 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.633575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.634944 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.634952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.635299 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.636149 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:19.636182 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:19.638675 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:19.642885 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.642895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.642917 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.643549 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.644016 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.644021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.644025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.644306 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.645105 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:19.645113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:19.645566 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.647306 (+ 1740us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.648143 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9058,"thread_start_us":8831,"threads_started":1}
I20260926 07:30:19.651844 15847 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:19.652204 15847 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.653144 15847 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:19.655479 15506 ts_manager.cc:194] Registered new tserver with Master: fe26fee40db34d7ba6d80ae8c879adbe (127.2.147.194:45113)
I20260926 07:30:19.657754 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:56521
W20260926 07:30:20.022989 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:20.023486 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:20.023681 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:20.035187 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:20.035432 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:20.035656 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:20.050045 15703 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:20.050509 15703 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:20.051925 15506 ts_manager.cc:194] Registered new tserver with Master: c0cd620420b247b29c3dbebe40ce2740 (127.2.147.193:35433)
I20260926 07:30:20.052079 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.147.212:35209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:33649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=36673
--tserver_master_addrs=127.2.147.254:37017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:20.053068 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:53407
I20260926 07:30:20.053960 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:20.055745 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:20.074090 15857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:20.074558 15858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:20.098382 15860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:20.098716 15851 server_base.cc:1034] running on GCE node
I20260926 07:30:20.099617 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:20.100827 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:20.105480 15851 hybrid_clock.cc:648] HybridClock initialized: now 1790407820105249 us; error 208 us; skew 500 ppm
I20260926 07:30:20.105958 15851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:20 dist-test-slave-1dwd krb5kdc[14495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:20.110474 15851 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:20.118629 15851 webserver.cc:460] Webserver started at http://127.2.147.195:36673/ using document root <none> and password file <none>
I20260926 07:30:20.119526 15851 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:20.119702 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:30:20 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
Sep 26 07:30:20 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:20.180598 15851 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.003s
I20260926 07:30:20.196028 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:20.200905 15851 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.006s sys 0.009s
I20260926 07:30:20.201229 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "30256e28286a409fb87e05fb0ddf1ade"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-1dwd"
server_key: "e51013cc631a0dbded85b3e70407061d"
server_key_iv: "a4a96b9ba3325489adfb40c0c1ab203a"
server_key_version: "kuduclusterkey@0"
I20260926 07:30:20.202993 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:20.227854 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:20.232141 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:20.232437 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:20.233485 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:20.234912 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:20.235033 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:20.235165 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:20.235266 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 07:30:20.288882 15851 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:33649
I20260926 07:30:20.288939 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:33649 every 8 connection(s)
I20260926 07:30:20.290839 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:20.295611 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 15851
Sep 26 07:30:20 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407820, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:20.335816 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.296916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.2.147.254:37017)
0926 07:30:20.315731 (+ 18815us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.315740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.316380 (+ 640us) server_negotiation.cc:407] Connection header received
0926 07:30:20.317516 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.317521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.317597 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.317721 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:20.319737 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.320797 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.322860 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.323552 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.323662 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.327340 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:20.327347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:20.327351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:20.327413 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:20.331003 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.331932 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.331939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.331948 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.332119 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.332941 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.332947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.332950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.333343 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:20.333612 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.335154 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.335405 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18634,"spinlock_wait_cycles":5888,"thread_start_us":10571,"threads_started":1}
I20260926 07:30:20.339095 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.295376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:47581 (local address 127.2.147.254:37017)
0926 07:30:20.307751 (+ 12375us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.307758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.317297 (+ 9539us) server_negotiation.cc:407] Connection header received
0926 07:30:20.320092 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.320098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.320172 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.320272 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:20.322531 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.323738 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.326371 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.327077 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.327184 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.331784 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:20.331791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:20.331795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:20.331843 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:20.335405 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.336322 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.336329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.336334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.336471 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.337034 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.337039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.337045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.337354 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:20.337464 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.338497 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.338714 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12223,"thread_start_us":20003,"threads_started":1}
I20260926 07:30:20.341135 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.294407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37017 (local address 127.2.147.195:47581)
0926 07:30:20.317072 (+ 22665us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.317131 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.319738 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.320786 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.320799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.321466 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:20.322288 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.322310 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.323934 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.323939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.326178 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.326196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.327609 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.327617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.327899 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.328604 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:20.328641 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:20.331482 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:20.335567 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:20.335577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:20.335596 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:20.336179 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:20.336630 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:20.336634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:20.336637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:20.336906 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:20.337595 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:20.337604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:20.338054 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.339158 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.339742 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22321,"thread_start_us":22100,"threads_started":1}
I20260926 07:30:20.343120 15981 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37017
I20260926 07:30:20.343510 15981 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:20.344437 15981 heartbeater.cc:511] Master 127.2.147.254:37017 requested a full tablet report, sending...
I20260926 07:30:20.346097 15506 ts_manager.cc:194] Registered new tserver with Master: 30256e28286a409fb87e05fb0ddf1ade (127.2.147.195:33649)
I20260926 07:30:20.346911 15506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:47581
I20260926 07:30:20.351292 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:20.374185 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.359171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.2.147.254:37017)
0926 07:30:20.359386 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.359395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.359932 (+ 537us) server_negotiation.cc:407] Connection header received
0926 07:30:20.360140 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.360146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.360222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.360326 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:20.362155 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.363404 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.364792 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.365250 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.365398 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.368037 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:20.368048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:20.368053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:20.368086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:20.370798 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.371752 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.371758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.371761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.371899 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:20.372436 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:20.372441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:20.372444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:20.372800 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:20.373036 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.373593 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.373830 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:30:20.380782 15506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:20.390815 15506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 07:30:20 dist-test-slave-1dwd krb5kdc[14495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for HTTP/127.2.147.212@KRBTEST.COM
I20260926 07:30:20.524716 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.447021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54838)
0926 07:30:20.471549 (+ 24528us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.471998 (+ 449us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.472268 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.514253 (+ 41985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.514263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.514306 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.514839 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.514858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.517064 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.517070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.519606 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.519621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.523675 (+ 4054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.523682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.523903 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.523968 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.524100 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.524309 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16007,"thread_start_us":68194,"threads_started":1}
I20260926 07:30:20.526463 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.447679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.2.147.193:35433)
0926 07:30:20.513526 (+ 65847us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.513535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.513567 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:30:20.513675 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.513684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.513969 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.514199 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.515093 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.516790 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.519957 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.522302 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.525590 (+ 3288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.525851 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.525937 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.526115 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65670,"thread_start_us":67881,"threads_started":1}
I20260926 07:30:20.534008 15638 tablet_service.cc:1467] Processing CreateTablet for tablet 69f55486084a4bb590367238d1b90b0a (DEFAULT_TABLE table=test-table [id=7e2cd971402441cfb9862ab032f67211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:20.535455 15638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f55486084a4bb590367238d1b90b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:20.536834 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.456747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:45113 (local address 127.0.0.1:37484)
0926 07:30:20.512822 (+ 56075us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.512844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.513023 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.518220 (+ 5197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.518225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.518250 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.518644 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.518656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.525147 (+ 6491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.525153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.527283 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.527300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.536019 (+ 8719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.536028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.536204 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.536239 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.536337 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.536484 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55937,"thread_start_us":58641,"threads_started":1}
I20260926 07:30:20.538291 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.457369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.2.147.194:45113)
0926 07:30:20.512401 (+ 55032us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.512410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.515873 (+ 3463us) server_negotiation.cc:407] Connection header received
0926 07:30:20.515974 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.515981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.516231 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.516417 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.523321 (+ 6904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.524956 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.534210 (+ 9254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.535759 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.535905 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.536110 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.537481 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.537692 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54862,"thread_start_us":57725,"threads_started":1}
I20260926 07:30:20.542539 15782 tablet_service.cc:1467] Processing CreateTablet for tablet 69f55486084a4bb590367238d1b90b0a (DEFAULT_TABLE table=test-table [id=7e2cd971402441cfb9862ab032f67211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:20.544010 15782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f55486084a4bb590367238d1b90b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:20.547256 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.462417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:33649 (local address 127.0.0.1:46380)
0926 07:30:20.514862 (+ 52445us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.514881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.515003 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.517606 (+ 2603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.517611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.517638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.518010 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.518025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.530432 (+ 12407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.530438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.532592 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.532609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.546402 (+ 13793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.546408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.546590 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.546624 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.546721 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.546873 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52304,"thread_start_us":740,"threads_started":1}
I20260926 07:30:20.548326 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.463009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.2.147.195:33649)
0926 07:30:20.511961 (+ 48952us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.511993 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.516697 (+ 4704us) server_negotiation.cc:407] Connection header received
0926 07:30:20.516792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.516817 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.517028 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.517208 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.527515 (+ 10307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.529194 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.536218 (+ 7024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.545848 (+ 9630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.546000 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.546238 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.547800 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.547982 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48731,"thread_start_us":70912,"threads_started":1}
I20260926 07:30:20.552196 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 69f55486084a4bb590367238d1b90b0a (DEFAULT_TABLE table=test-table [id=7e2cd971402441cfb9862ab032f67211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:20.554518 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f55486084a4bb590367238d1b90b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:20.570333 16004 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Bootstrap starting.
I20260926 07:30:20.574910 16005 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Bootstrap starting.
I20260926 07:30:20.575728 16004 tablet_bootstrap.cc:654] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:20.577190 16004 log.cc:824] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:20.578683 16006 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Bootstrap starting.
I20260926 07:30:20.580168 16005 tablet_bootstrap.cc:654] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:20.581601 16005 log.cc:824] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:20.585098 16006 tablet_bootstrap.cc:654] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:20.589068 16004 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: No bootstrap required, opened a new log
I20260926 07:30:20.589488 16004 ts_tablet_manager.cc:1397] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260926 07:30:20.597105 16006 log.cc:824] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:20.600919 16005 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: No bootstrap required, opened a new log
I20260926 07:30:20.601305 16005 ts_tablet_manager.cc:1397] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.008s
I20260926 07:30:20.606829 16004 raft_consensus.cc:348] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.607347 16004 raft_consensus.cc:374] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:20.607474 16004 raft_consensus.cc:729] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26fee40db34d7ba6d80ae8c879adbe, State: Initialized, Role: FOLLOWER
I20260926 07:30:20.608273 16004 consensus_queue.cc:260] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.609519 15847 heartbeater.cc:503] Master 127.2.147.254:37017 was elected leader, sending a full tablet report...
I20260926 07:30:20.610365 16004 ts_tablet_manager.cc:1428] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Time spent starting tablet: real 0.021s user 0.015s sys 0.000s
I20260926 07:30:20.611668 16006 tablet_bootstrap.cc:492] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: No bootstrap required, opened a new log
I20260926 07:30:20.612025 16006 ts_tablet_manager.cc:1397] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20260926 07:30:20.616360 16005 raft_consensus.cc:348] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.616801 16005 raft_consensus.cc:374] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:20.616897 16005 raft_consensus.cc:729] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30256e28286a409fb87e05fb0ddf1ade, State: Initialized, Role: FOLLOWER
I20260926 07:30:20.617501 16005 consensus_queue.cc:260] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.618683 15981 heartbeater.cc:503] Master 127.2.147.254:37017 was elected leader, sending a full tablet report...
W20260926 07:30:20.619059 15848 tablet.cc:2369] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:20.620174 16005 ts_tablet_manager.cc:1428] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260926 07:30:20.622983 16006 raft_consensus.cc:348] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.623423 16006 raft_consensus.cc:374] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:20.623508 16006 raft_consensus.cc:729] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cd620420b247b29c3dbebe40ce2740, State: Initialized, Role: FOLLOWER
I20260926 07:30:20.624059 16006 consensus_queue.cc:260] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.625130 15703 heartbeater.cc:503] Master 127.2.147.254:37017 was elected leader, sending a full tablet report...
I20260926 07:30:20.625697 16006 ts_tablet_manager.cc:1428] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260926 07:30:20.760227 15704 tablet.cc:2369] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:30:20.803442 15982 tablet.cc:2369] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:20.850384 16011 raft_consensus.cc:482] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:20.850728 16011 raft_consensus.cc:504] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.852794 16011 leader_election.cc:288] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cd620420b247b29c3dbebe40ce2740 (127.2.147.193:35433), fe26fee40db34d7ba6d80ae8c879adbe (127.2.147.194:45113)
I20260926 07:30:20.874173 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.854099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:36997 (local address 127.2.147.193:35433)
0926 07:30:20.854296 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.854319 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.864056 (+ 9737us) server_negotiation.cc:407] Connection header received
0926 07:30:20.864621 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.864627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.864710 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.864775 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.866160 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.867675 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.870994 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.872736 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.872921 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.873049 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.873760 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.873902 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:30:20.882408 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.853099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.2.147.195:36997)
0926 07:30:20.863900 (+ 10801us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.863923 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.864417 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.864979 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.864983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.865087 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.865834 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.865853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.867937 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.867946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.870734 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.870758 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.873357 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.873365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.873557 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.873594 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.874965 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.875120 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10639,"thread_start_us":19999,"threads_started":1}
I20260926 07:30:20.882601 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.853878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:53927 (local address 127.2.147.194:45113)
0926 07:30:20.854067 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.854074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.873714 (+ 19640us) server_negotiation.cc:407] Connection header received
0926 07:30:20.874002 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.874010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.874117 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.874185 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.875141 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.876438 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.879462 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.881040 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.881214 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.881321 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.882091 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.882274 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:30:20.883831 15658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f55486084a4bb590367238d1b90b0a" candidate_uuid: "30256e28286a409fb87e05fb0ddf1ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cd620420b247b29c3dbebe40ce2740" is_pre_election: true
I20260926 07:30:20.884501 15658 raft_consensus.cc:2454] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30256e28286a409fb87e05fb0ddf1ade in term 0.
I20260926 07:30:20.884773 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.853482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:45113 (local address 127.2.147.195:53927)
0926 07:30:20.873591 (+ 20109us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.873615 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.873830 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.874412 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.874419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.874450 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.874922 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.874937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.876644 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.876652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.879238 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.879260 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.881573 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.881581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.881838 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.881889 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.883106 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.884394 (+ 1288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19937,"thread_start_us":10609,"threads_started":1}
I20260926 07:30:20.885349 15872 leader_election.cc:302] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30256e28286a409fb87e05fb0ddf1ade, c0cd620420b247b29c3dbebe40ce2740; no voters:
I20260926 07:30:20.885792 16011 raft_consensus.cc:2781] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:20.885919 16011 raft_consensus.cc:482] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:20.886039 16011 raft_consensus.cc:3037] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:20.885898 15802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f55486084a4bb590367238d1b90b0a" candidate_uuid: "30256e28286a409fb87e05fb0ddf1ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" is_pre_election: true
I20260926 07:30:20.886456 15802 raft_consensus.cc:2454] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30256e28286a409fb87e05fb0ddf1ade in term 0.
I20260926 07:30:20.889153 16011 raft_consensus.cc:504] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.889998 16011 leader_election.cc:288] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [CANDIDATE]: Term 1 election: Requested vote from peers c0cd620420b247b29c3dbebe40ce2740 (127.2.147.193:35433), fe26fee40db34d7ba6d80ae8c879adbe (127.2.147.194:45113)
I20260926 07:30:20.890588 15658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f55486084a4bb590367238d1b90b0a" candidate_uuid: "30256e28286a409fb87e05fb0ddf1ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cd620420b247b29c3dbebe40ce2740"
I20260926 07:30:20.890829 15658 raft_consensus.cc:3037] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:20.892238 15802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f55486084a4bb590367238d1b90b0a" candidate_uuid: "30256e28286a409fb87e05fb0ddf1ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe26fee40db34d7ba6d80ae8c879adbe"
I20260926 07:30:20.892449 15802 raft_consensus.cc:3037] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:20.894827 15658 raft_consensus.cc:2454] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30256e28286a409fb87e05fb0ddf1ade in term 1.
I20260926 07:30:20.895253 15802 raft_consensus.cc:2454] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30256e28286a409fb87e05fb0ddf1ade in term 1.
I20260926 07:30:20.895382 15872 leader_election.cc:302] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30256e28286a409fb87e05fb0ddf1ade, c0cd620420b247b29c3dbebe40ce2740; no voters:
I20260926 07:30:20.895758 16011 raft_consensus.cc:2781] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:20.896463 16011 raft_consensus.cc:686] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 1 LEADER]: Becoming Leader. State: Replica: 30256e28286a409fb87e05fb0ddf1ade, State: Running, Role: LEADER
I20260926 07:30:20.897197 16011 consensus_queue.cc:237] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } }
I20260926 07:30:20.903196 15506 catalog_manager.cc:5495] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 30256e28286a409fb87e05fb0ddf1ade (127.2.147.195). New cstate: current_term: 1 leader_uuid: "30256e28286a409fb87e05fb0ddf1ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30256e28286a409fb87e05fb0ddf1ade" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 33649 } health_report { overall_health: HEALTHY } } }
I20260926 07:30:20.992333 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.986514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.2.147.195:33649)
0926 07:30:20.986758 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.986765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.987057 (+ 292us) server_negotiation.cc:407] Connection header received
0926 07:30:20.987321 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.987325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.987408 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.987567 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:20.988010 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.988740 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.990107 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.990506 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.990649 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.991163 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:20.991736 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:20.991895 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.991989 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.992071 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:30:21.000293 15658 raft_consensus.cc:1261] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 1 FOLLOWER]: Refusing update from remote peer 30256e28286a409fb87e05fb0ddf1ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:21.000332 15802 raft_consensus.cc:1261] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 1 FOLLOWER]: Refusing update from remote peer 30256e28286a409fb87e05fb0ddf1ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:21.001298 16022 consensus_queue.cc:1035] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cd620420b247b29c3dbebe40ce2740" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:21.001677 16011 consensus_queue.cc:1035] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe26fee40db34d7ba6d80ae8c879adbe" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:21.013872 16027 mvcc.cc:205] Tried to move back new op lower bound from 7333510434807451648 to 7333510434400841728. Current Snapshot: MvccSnapshot[applied={T|T < 7333510434807451648}]
I20260926 07:30:21.032229 15506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:21.032512 15506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:21.035917 15506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c52cf71f51244ca499300b41175af085: Sending DeleteTablet for 3 replicas of tablet 69f55486084a4bb590367238d1b90b0a
I20260926 07:30:21.036974 15638 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f55486084a4bb590367238d1b90b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:54838
I20260926 07:30:21.037053 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f55486084a4bb590367238d1b90b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:46380
I20260926 07:30:21.037089 15782 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f55486084a4bb590367238d1b90b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:21 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:37484
I20260926 07:30:21.043061 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15543
I20260926 07:30:21.057950 16038 tablet_replica.cc:330] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: stopping tablet replica
I20260926 07:30:21.058441 16038 raft_consensus.cc:2229] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:21.058962 16038 raft_consensus.cc:2258] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:21.059597 16037 tablet_replica.cc:330] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: stopping tablet replica
I20260926 07:30:21.060179 16036 tablet_replica.cc:330] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: stopping tablet replica
I20260926 07:30:21.060776 16037 raft_consensus.cc:2229] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:21.061543 16037 raft_consensus.cc:2258] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:21.062762 16038 ts_tablet_manager.cc:1905] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:21.063640 16037 ts_tablet_manager.cc:1905] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:21.069226 16036 raft_consensus.cc:2229] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:21.069900 16036 raft_consensus.cc:2258] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:21.070914 16037 ts_tablet_manager.cc:1918] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:21.071131 16037 log.cc:1196] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/69f55486084a4bb590367238d1b90b0a
I20260926 07:30:21.071280 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.044672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.2.147.193:35433)
0926 07:30:21.044867 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:21.044873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:21.047970 (+ 3097us) server_negotiation.cc:407] Connection header received
0926 07:30:21.048910 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:21.048919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:21.049039 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:21.049129 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:21.054137 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.055787 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.059024 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.059741 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.059874 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:21.063620 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:21.063637 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:21.063659 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:21.063711 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:21.067642 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.068554 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.068564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.068567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.068726 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.069215 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.069220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.069223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.069699 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:21.070013 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:21.070617 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 07:30:21.070854 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:21.071743 16037 ts_tablet_manager.cc:1939] T 69f55486084a4bb590367238d1b90b0a P 30256e28286a409fb87e05fb0ddf1ade: Deleting consensus metadata
I20260926 07:30:21.072781 15698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:21.073557 15492 catalog_manager.cc:4841] TS 30256e28286a409fb87e05fb0ddf1ade (127.2.147.195:33649): tablet 69f55486084a4bb590367238d1b90b0a (table test-table [id=7e2cd971402441cfb9862ab032f67211]) successfully deleted
I20260926 07:30:21.075567 16038 ts_tablet_manager.cc:1918] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:21.075996 16038 log.cc:1196] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/69f55486084a4bb590367238d1b90b0a
I20260926 07:30:21.076604 16038 ts_tablet_manager.cc:1939] T 69f55486084a4bb590367238d1b90b0a P fe26fee40db34d7ba6d80ae8c879adbe: Deleting consensus metadata
I20260926 07:30:21.078437 15490 catalog_manager.cc:4841] TS fe26fee40db34d7ba6d80ae8c879adbe (127.2.147.194:45113): tablet 69f55486084a4bb590367238d1b90b0a (table test-table [id=7e2cd971402441cfb9862ab032f67211]) successfully deleted
I20260926 07:30:21.073100 16036 ts_tablet_manager.cc:1905] T 69f55486084a4bb590367238d1b90b0a P c0cd620420b247b29c3dbebe40ce2740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:21.167917 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15543
W20260926 07:30:21.186532 15493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.147.193:35433 (error 108)
I20260926 07:30:21.186666 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15707
I20260926 07:30:21.188905 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.188115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54852)
0926 07:30:21.188574 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.188687 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":208,"threads_started":1}
W20260926 07:30:21.189129 15493 catalog_manager.cc:4584] TS c0cd620420b247b29c3dbebe40ce2740 (127.2.147.193:35433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69f55486084a4bb590367238d1b90b0a: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
I20260926 07:30:21.202648 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.187489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.2.147.194:45113)
0926 07:30:21.187676 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:21.187683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:21.187964 (+ 281us) server_negotiation.cc:407] Connection header received
0926 07:30:21.188173 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:21.188179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:21.188296 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:21.188409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:21.190728 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.191480 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.193300 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.193683 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.193786 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:21.196935 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:21.196949 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:21.196964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:21.196996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:21.199825 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.200656 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.200668 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.200675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.200820 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.201218 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.201221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.201223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.201554 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:21.201821 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:21.202156 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:30:21.202320 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:30:21.203408 15842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:21.242820 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.242247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54858)
0926 07:30:21.242512 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.242642 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 07:30:21.243971 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.243564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54872)
0926 07:30:21.243733 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.243817 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:30:21.304705 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.304159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54882)
0926 07:30:21.304438 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.304545 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 07:30:21.304978 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15707
I20260926 07:30:21.306833 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.306435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54886)
0926 07:30:21.306618 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.306692 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 07:30:21.331177 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15851
I20260926 07:30:21.351256 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.332569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.2.147.195:33649)
0926 07:30:21.332755 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:21.332768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:21.332794 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:30:21.332871 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:21.332878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:21.333012 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:21.333135 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:21.335425 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.336804 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.338809 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:21.339749 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:21.339912 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:21.343860 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:21.343875 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:21.343898 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:21.343965 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:21.347993 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.348955 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.348963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.348965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.349140 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:21.349589 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:21.349596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:21.349598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:21.349951 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:21.350196 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:21.350554 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 07:30:21.350805 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:30:21.352236 15976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:21.388051 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.387572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54896)
0926 07:30:21.387799 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.387896 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:30:21.389183 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.388792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54906)
0926 07:30:21.388975 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.389042 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 07:30:21.476351 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15851
I20260926 07:30:21.495524 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 15473
I20260926 07:30:21.496690 15536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:21.610585 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.609831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:35433 (local address 127.0.0.1:54918)
0926 07:30:21.610187 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.610341 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:35433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:30:21.611084 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 15473
I20260926 07:30:21.635802 2639 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:30:21.971490 2639 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:30:21.971661 2639 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:30:22.316802 2639 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:30:22.317 UTC [14525] LOG: received smart shutdown request
2026-09-26 07:30:22.320 UTC [14525] LOG: background worker "logical replication launcher" (PID 14534) exited with exit code 1
2026-09-26 07:30:22.320 UTC [14529] LOG: shutting down
2026-09-26 07:30:22.490 UTC [14525] LOG: database system is shut down
2026-09-26T07:30:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53073 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:22 dist-test-slave-1dwd krb5kdc[16054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:22 dist-test-slave-1dwd krb5kdc[16054](info): set up 2 sockets
Sep 26 07:30:22 dist-test-slave-1dwd krb5kdc[16054](info): commencing operation
krb5kdc: starting...
W20260926 07:30:23.719841 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:30:23 dist-test-slave-1dwd krb5kdc[16054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:30:23.957052 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38909
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:24.288208 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:24.288497 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:24.288664 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:24.295857 16067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:24.295989 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:24.296074 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:24.296176 16067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:24.296244 16067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:24.305969 16067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:24.307415 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:24.308755 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:24.320758 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.320739 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.320808 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:24.321833 16067 server_base.cc:1034] running on GCE node
I20260926 07:30:24.322563 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:24.324003 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:24.325240 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790407824325218 us; error 57 us; skew 500 ppm
I20260926 07:30:24.325596 16067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:24 dist-test-slave-1dwd krb5kdc[16054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:24.332844 16067 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:24.334444 16067 webserver.cc:460] Webserver started at http://127.2.147.254:43653/ using document root <none> and password file <none>
I20260926 07:30:24.335155 16067 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:24.335340 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:24.335655 16067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:24.338632 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "01dba1c2d451498d88cf5b8934fa6f21"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "927b89a5c714bea1382b0879a4734905"
server_key_iv: "f20a45dd38a41317ac6fa257a3db83cc"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.339550 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "01dba1c2d451498d88cf5b8934fa6f21"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "927b89a5c714bea1382b0879a4734905"
server_key_iv: "f20a45dd38a41317ac6fa257a3db83cc"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.344678 16067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:30:24.348296 16082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.349802 16067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:24.349998 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "01dba1c2d451498d88cf5b8934fa6f21"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "927b89a5c714bea1382b0879a4734905"
server_key_iv: "f20a45dd38a41317ac6fa257a3db83cc"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.350231 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:24.371582 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:24.375586 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:24.375895 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:24.394353 16067 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38909
I20260926 07:30:24.394380 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38909 every 8 connection(s)
I20260926 07:30:24.395776 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:24.399410 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:24.400599 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16067
I20260926 07:30:24.400841 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:30:24.401456 2639 external_mini_cluster.cc:1428] Setting key b851a38fed3e948b120122538e59632f
I20260926 07:30:24.405990 16134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:24 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.253@KRBTEST.COM" created.
I20260926 07:30:24.430027 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.409224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38909 (local address 127.0.0.1:41662)
0926 07:30:24.410206 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.410274 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.412679 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.413575 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.413580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.414130 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.414470 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.414486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.416790 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.416794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.417863 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.417875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.419295 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.419301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.419422 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.420178 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.420204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.422509 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.426093 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.426102 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.426111 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.426490 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.426973 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.426976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.426979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.427118 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.428128 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.428134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.428328 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.428539 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.428917 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"spinlock_wait_cycles":29440,"thread_start_us":393,"threads_started":1}
I20260926 07:30:24.430078 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.409218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.2.147.254:38909)
0926 07:30:24.410857 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.410870 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.410921 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:30:24.413038 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.413048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.413346 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.413565 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.414664 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.416630 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.418010 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.419019 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.419246 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.422764 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.422772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.422784 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.422815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.425950 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.426629 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.426641 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.426647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.426860 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.427246 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.427254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.427261 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.427989 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.428205 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.428725 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.429110 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":1399,"spinlock_wait_cycles":43904,"thread_start_us":1197,"threads_started":1}
I20260926 07:30:24.433292 16134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:30:24.434386 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.433994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43562)
0926 07:30:24.434165 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.434272 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:30:24.434625 16085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
I20260926 07:30:24.436267 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.435911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43572)
0926 07:30:24.436066 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.436135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:30:24.436491 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:33997: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
I20260926 07:30:24.455435 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:33997
--webserver_interface=127.2.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:24.485836 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } attempt: 1
I20260926 07:30:24.487457 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.486853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43578)
0926 07:30:24.487070 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.487268 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:30:24.488897 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.488543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43584)
0926 07:30:24.488679 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.488773 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:30:24.489217 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:33997: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
I20260926 07:30:24.557502 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } attempt: 2
I20260926 07:30:24.558813 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.558307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43598)
0926 07:30:24.558564 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.558665 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:30:24.560024 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.559664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43608)
0926 07:30:24.559822 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.559904 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 07:30:24.560292 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:33997: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
I20260926 07:30:24.651602 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } attempt: 3
I20260926 07:30:24.652877 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.652365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43612)
0926 07:30:24.652577 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.652690 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 07:30:24.654036 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.653613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43616)
0926 07:30:24.653803 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.653900 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 07:30:24.654232 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:33997: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
I20260926 07:30:24.797475 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } attempt: 4
I20260926 07:30:24.799005 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.798264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43628)
0926 07:30:24.798679 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.798805 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260926 07:30:24.800165 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.799735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43638)
0926 07:30:24.799899 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.800009 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:30:24.800362 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:33997: Network error: Client connection negotiation failed: client connection to 127.2.147.253:33997: connect: Connection refused (error 111)
W20260926 07:30:24.806301 16141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:24.806602 16141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:24.806761 16141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:24.814154 16141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:24.814271 16141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:24.814545 16141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:24.814662 16141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:24.814741 16141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:24.825459 16141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:24.826853 16141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:24.828195 16141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:24.840188 16149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.840241 16146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.840152 16147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:24.840920 16141 server_base.cc:1034] running on GCE node
I20260926 07:30:24.841768 16141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:24.843186 16141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:24.844473 16141 hybrid_clock.cc:648] HybridClock initialized: now 1790407824844422 us; error 49 us; skew 500 ppm
I20260926 07:30:24.844866 16141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:24 dist-test-slave-1dwd krb5kdc[16054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:24.848176 16141 init.cc:377] Logged in from keytab as kudu/127.2.147.253@KRBTEST.COM (short username kudu)
I20260926 07:30:24.849671 16141 webserver.cc:460] Webserver started at http://127.2.147.253:38137/ using document root <none> and password file <none>
I20260926 07:30:24.850430 16141 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:24.850574 16141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:24.850911 16141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:24.854199 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data/instance:
uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "7740c4fd7737e4fdc54b4a7a67627a38"
server_key_iv: "d1bdac3faacc19c34da0db8caee135c4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.855106 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal/instance:
uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "7740c4fd7737e4fdc54b4a7a67627a38"
server_key_iv: "d1bdac3faacc19c34da0db8caee135c4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.860670 16141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:30:24.864075 16156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.865466 16141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:24.865610 16141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-1dwd"
server_key: "7740c4fd7737e4fdc54b4a7a67627a38"
server_key_iv: "d1bdac3faacc19c34da0db8caee135c4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.865777 16141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:24.881949 16141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:24.886523 16141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:24.886812 16141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:24.904948 16141 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.253:33997
I20260926 07:30:24.904971 16207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.253:33997 every 8 connection(s)
I20260926 07:30:24.906390 16141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
I20260926 07:30:24.907545 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16141
I20260926 07:30:24.907732 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-1/wal/instance
I20260926 07:30:24.908161 2639 external_mini_cluster.cc:1428] Setting key 5d6aeed75d1dced7ef6160504d485012
I20260926 07:30:24.910281 16208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:24.918623 16208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.252@KRBTEST.COM" created.
Sep 26 07:30:24 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:24.941771 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.922106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.2.147.254:38909)
0926 07:30:24.922346 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.922352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.923641 (+ 1289us) server_negotiation.cc:407] Connection header received
0926 07:30:24.925747 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.925757 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.925866 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.926005 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.927586 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.928737 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.930503 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.930914 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.931008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.934722 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.934731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.934740 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.934789 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.938548 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.939396 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.939401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.939403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.939528 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.939983 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.939995 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.939997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.940281 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.940440 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.941167 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.941437 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 07:30:24.942932 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.922568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38909 (local address 127.0.0.1:41670)
0926 07:30:24.923453 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.923501 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.925441 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.926236 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.926244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.926755 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.927405 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.927424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.929000 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.929004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.930366 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.930383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.931134 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.931139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.931416 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.932279 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.932302 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.934445 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.938705 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.938712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.938724 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.939187 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.939661 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.939665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.939669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.939857 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.940411 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.940418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.940774 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.941050 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.941595 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":378,"threads_started":1}
I20260926 07:30:24.944779 16208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:24 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
I20260926 07:30:24.961812 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.945629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43648)
0926 07:30:24.945826 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.945847 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.945954 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.946994 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.946998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.947499 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.947639 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.947648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.949055 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.949063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.949910 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.949920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.950769 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.950774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.951007 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.951700 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.951716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.955527 (+ 3811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.959295 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.959299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.959302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.959710 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.960039 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.960042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.960044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.960210 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.960902 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.960906 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.961013 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.961189 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.961457 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260926 07:30:24.962113 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.945897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.2.147.253:33997)
0926 07:30:24.946574 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.946579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.946607 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:30:24.946650 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.946657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.946841 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.947003 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.947801 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.948912 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.950050 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.950619 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.950758 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.955750 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.955761 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.955775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.955805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.959098 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.959812 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.959819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.959821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.959941 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.960335 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.960339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.960341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.960766 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.960962 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.961380 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.961687 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":530,"thread_start_us":397,"threads_started":1}
I20260926 07:30:24.962617 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:44005
--webserver_interface=127.2.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:24.964955 16208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:30:24.966465 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.965750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43792)
0926 07:30:24.966045 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.966255 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 07:30:24.966950 16160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:24.969120 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.968742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43806)
0926 07:30:24.968919 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.969012 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 07:30:24.969321 16208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.018632 16208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 1
I20260926 07:30:25.019896 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.019423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43810)
0926 07:30:25.019639 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.019749 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:30:25.020882 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.020545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43824)
0926 07:30:25.020718 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.020782 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:30:25.021065 16208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.089373 16208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 2
I20260926 07:30:25.090654 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.090070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43834)
0926 07:30:25.090319 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.090438 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 07:30:25.091830 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.091341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43846)
0926 07:30:25.091533 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.091658 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:30:25.092073 16208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.099601 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } attempt: 5
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:30:25.116706 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.100343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43650)
0926 07:30:25.100564 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.100586 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.100709 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.101083 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.101089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.101477 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.101657 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.101665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.103166 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.103173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.104865 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.104885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.105993 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.106002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.106389 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.107381 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.107408 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.110291 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.114210 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.114214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.114217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.114570 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.115011 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.115014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.115017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.115144 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.115899 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.115904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.116083 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.116217 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.116447 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:30:25.116990 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.100516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.2.147.253:33997)
0926 07:30:25.100712 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.100718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.100743 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:30:25.100822 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.100828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.100905 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.101036 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.101822 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.102979 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.105046 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.105800 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.105938 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.110569 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.110578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.110583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.110647 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.114030 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.114698 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.114706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.114712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.114885 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.115285 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.115292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.115296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.115758 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.115935 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.116405 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.116649 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:30:25.118062 16134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:30:25.119306 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.118879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43854)
0926 07:30:25.119041 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.119115 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:30:25.120249 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.119931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43864)
0926 07:30:25.120069 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.120153 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:30:25.120463 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.171660 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 1
I20260926 07:30:25.172881 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.172417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43876)
0926 07:30:25.172617 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.172718 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:30:25.173894 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.173553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43892)
0926 07:30:25.173719 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.173797 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:30:25.174099 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.183320 16208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 3
I20260926 07:30:25.184695 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.184036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43898)
0926 07:30:25.184303 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.184446 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 07:30:25.185856 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.185369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43912)
0926 07:30:25.185609 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.185710 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260926 07:30:25.186087 16208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.242331 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 2
I20260926 07:30:25.243582 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.243037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43922)
0926 07:30:25.243315 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.243403 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 07:30:25.244658 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.244313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43938)
0926 07:30:25.244466 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.244545 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:30:25.244892 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
W20260926 07:30:25.302157 16215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:25.302541 16215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:25.302713 16215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:25.310117 16215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:25.310278 16215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:25.310391 16215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:25.310495 16215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:25.310581 16215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:25.320492 16215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38909,127.2.147.253:33997,127.2.147.252:44005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:44005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:25.321930 16215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:25.323355 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:25.329299 16208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 4
I20260926 07:30:25.330470 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.329929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43940)
0926 07:30:25.330181 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.330280 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:30:25.331445 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.331071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43946)
0926 07:30:25.331246 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.331325 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:30:25.331635 16208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
W20260926 07:30:25.336083 16220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:25.336057 16223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:25.336138 16221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:25.337256 16215 server_base.cc:1034] running on GCE node
I20260926 07:30:25.338129 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:25.339605 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:25.340803 16215 hybrid_clock.cc:648] HybridClock initialized: now 1790407825340761 us; error 32 us; skew 500 ppm
I20260926 07:30:25.341164 16215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:25.344252 16215 init.cc:377] Logged in from keytab as kudu/127.2.147.252@KRBTEST.COM (short username kudu)
I20260926 07:30:25.345744 16215 webserver.cc:460] Webserver started at http://127.2.147.252:46783/ using document root <none> and password file <none>
I20260926 07:30:25.346449 16215 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:25.346585 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:25.346903 16215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:25.350063 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data/instance:
uuid: "2d6d43970c8a4015b871b3cb7571a101"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-1dwd"
server_key: "d99ffab6f8e2839bdc6017ae7e9fd0e7"
server_key_iv: "9aec5ecf693ba29cdf53c6a8d09b0f90"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.350914 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal/instance:
uuid: "2d6d43970c8a4015b871b3cb7571a101"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-1dwd"
server_key: "d99ffab6f8e2839bdc6017ae7e9fd0e7"
server_key_iv: "9aec5ecf693ba29cdf53c6a8d09b0f90"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.351095 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 3
I20260926 07:30:25.352255 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.351703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43962)
0926 07:30:25.351933 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.352048 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:30:25.353395 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.352928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43976)
0926 07:30:25.353120 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.353221 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 07:30:25.353652 16134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:44005: Network error: Client connection negotiation failed: client connection to 127.2.147.252:44005: connect: Connection refused (error 111)
I20260926 07:30:25.356937 16215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 07:30:25.360437 16230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:25.361763 16215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:25.361943 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "2d6d43970c8a4015b871b3cb7571a101"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-1dwd"
server_key: "d99ffab6f8e2839bdc6017ae7e9fd0e7"
server_key_iv: "9aec5ecf693ba29cdf53c6a8d09b0f90"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.362154 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:25.381943 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:25.386107 16215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:25.386401 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:25.405010 16215 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.252:44005
I20260926 07:30:25.405042 16281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.252:44005 every 8 connection(s)
I20260926 07:30:25.406443 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
I20260926 07:30:25.410084 16282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.416599 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16215
I20260926 07:30:25.416821 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407623164223-2639-0/minicluster-data/master-2/wal/instance
I20260926 07:30:25.416342 16282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:30:25.417306 2639 external_mini_cluster.cc:1428] Setting key f3b5d09cd2c8a9b1f64a3d8454b5facd
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:30:25.438230 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.420238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.2.147.254:38909)
0926 07:30:25.420430 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.420436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.421043 (+ 607us) server_negotiation.cc:407] Connection header received
0926 07:30:25.422079 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.422084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.422165 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.422285 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.424000 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.425140 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.427178 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.427731 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.427853 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.431859 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.431866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.431871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.431915 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.434802 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.435695 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.435699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.435701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.435834 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.436327 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.436334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.436338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.436672 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.436850 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.437691 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.437864 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:25.439491 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.420162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38909 (local address 127.0.0.1:41678)
0926 07:30:25.420883 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.420933 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.421827 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.422375 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.422382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.423228 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.423812 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.423837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.425296 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.425299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.427034 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.427048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.427942 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.427947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.428168 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.428892 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.428931 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.431618 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.434937 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.434949 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.434968 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.435570 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.435949 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.435956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.435960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.436210 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.436806 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.436812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.437288 (+ 476us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.437584 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.438109 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":66,"thread_start_us":330,"threads_started":1}
I20260926 07:30:25.439527 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.419106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.2.147.252:44005)
0926 07:30:25.419843 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.419853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.419889 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:30:25.421815 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.421837 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.422146 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.422347 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.424149 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.425788 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.427597 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.428309 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.428405 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.432889 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.432903 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.432922 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.432951 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.436126 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.436941 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.436948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.436951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.437095 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.437450 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.437455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.437458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.437871 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.438173 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.438941 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.439110 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":546,"thread_start_us":310,"threads_started":1}
I20260926 07:30:25.442948 16282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:25.458532 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.443764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.2.147.253:33997)
0926 07:30:25.443951 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.443955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.443972 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:30:25.444151 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.444154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.444213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.444288 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.445107 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.446117 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.447798 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.448442 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.448571 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.452291 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.452296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.452299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.452335 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.455330 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.456129 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.456132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.456134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.456259 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.456666 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.456669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.456671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.456910 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.457043 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.458054 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.458263 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:30:25.458573 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.443723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:33997 (local address 127.0.0.1:43666)
0926 07:30:25.443908 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.443927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.444029 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.444358 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.444364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.444803 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.444971 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.444979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.446251 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.446256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.447648 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.447664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.448594 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.448600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.448831 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.449487 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.449510 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.452077 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.455478 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.455485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.455489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.456015 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.456372 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.456377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.456381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.456560 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.457609 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.457614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.457783 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.457947 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.458225 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:30:25.459968 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.442635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.2.147.254:38909)
0926 07:30:25.442821 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.442825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.442876 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:30:25.443048 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.443052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.443132 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.443274 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.444883 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.445988 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.447799 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.448442 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.448571 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.453519 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.453524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.453528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.453559 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.456905 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.457575 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.457581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.457585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.457745 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.458165 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.458172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.458176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.458508 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.458843 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.459407 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.459616 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:30:25.460099 16282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:30:25.473851 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.460886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.2.147.252:44005)
0926 07:30:25.461069 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.461074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.461303 (+ 229us) server_negotiation.cc:407] Connection header received
0926 07:30:25.461460 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.461465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.461545 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.461659 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.462721 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.463478 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.464688 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.465194 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.465282 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.468664 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.468669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.468672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.468699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.471469 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.472050 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.472058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.472062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.472184 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.472517 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.472524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.472528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.472795 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.472960 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.473269 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.473479 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:30:25.474023 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.460950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:43998)
0926 07:30:25.461179 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.461229 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.461340 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.461878 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.461884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.462382 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.462580 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.462593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.463601 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.463610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.464558 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.464574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.465309 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.465314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.465503 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.466139 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.466160 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.468477 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.471588 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.471595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.471599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.471927 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.472287 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.472293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.472297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.472418 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.472903 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.472908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.473059 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.473199 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.473639 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":134}
I20260926 07:30:25.476668 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.461786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.2.147.253:33997)
0926 07:30:25.461938 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.461942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.461956 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:25.462092 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.462095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.462145 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.462223 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.463667 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.464348 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.465963 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.466484 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.466574 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.471298 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.471304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.471309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.471353 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.473992 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.474787 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.474794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.474798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.474944 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.475387 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.475390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.475392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.475655 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.475860 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.476248 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.476408 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:30:25.479315 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: Bootstrap starting.
I20260926 07:30:25.482908 16282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.483850 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:25.484133 16282 log.cc:824] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.486852 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: No bootstrap required, opened a new log
I20260926 07:30:25.498862 16282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.500113 16282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.500248 16282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6d43970c8a4015b871b3cb7571a101, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.501098 16282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.508086 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.508540 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.510466 16282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:25.515491 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.489218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.2.147.253:33997)
0926 07:30:25.489407 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.489411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.497207 (+ 7796us) server_negotiation.cc:407] Connection header received
0926 07:30:25.497369 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.497378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.497490 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.497607 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.500394 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.501579 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.503689 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.504354 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.504453 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.508258 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.508265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.508271 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.508320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.512017 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.512929 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.512935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.512938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.513096 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.513557 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.513562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.513565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.513970 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.514214 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.514844 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.515085 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:30:25.517433 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.489190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.2.147.254:38909)
0926 07:30:25.489387 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.489393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.493530 (+ 4137us) server_negotiation.cc:407] Connection header received
0926 07:30:25.494015 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.494024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.494111 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.494200 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.497383 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.498478 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.501089 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.501849 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.501986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.507376 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.507386 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.507394 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.507466 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.511858 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.513338 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.513348 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.513354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.513570 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.514094 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.514102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.514109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.514571 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.514867 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.515916 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.516279 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:30:25.518245 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.489301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.2.147.252:44005)
0926 07:30:25.489478 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.489487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.499794 (+ 10307us) server_negotiation.cc:407] Connection header received
0926 07:30:25.499933 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.499938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.500018 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.500142 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.502334 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.503417 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.505576 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.506368 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.506503 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.509927 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.509933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.509937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.509974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.514386 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.515254 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.515260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.515264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.515435 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.515885 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.515890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.515893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.516301 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.516534 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.517417 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.517627 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:30:25.530665 16306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:25.530838 16306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:25.530869 16134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 4
I20260926 07:30:25.525585 16215 master_runner.cc:186] Error getting master registration for 127.2.147.254:38909: OK, instance_id { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" instance_seqno: 1790407824389044 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:30:25.535631 16215 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:30:25.547875 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.531428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:44006)
0926 07:30:25.531716 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.531737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.531842 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.532410 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.532416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.532920 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.533139 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.533150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.535318 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.535324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.536462 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.536474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.537435 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.537439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.537575 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.538213 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.538230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.540539 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.545387 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.545391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.545393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.545762 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.546235 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.546239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.546241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.546405 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.547050 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.547053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.547189 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.547382 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.547610 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 07:30:25.548127 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.531489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.2.147.252:44005)
0926 07:30:25.532083 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.532089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.532105 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:25.532163 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.532168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.532242 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.532350 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.533305 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.534457 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.536600 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.537297 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.537417 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.540739 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.540747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.540753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.540796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.545244 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.545900 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.545906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.545910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.546077 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.546543 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.546549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.546552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.546921 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.547083 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.547505 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.547746 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260926 07:30:25.554157 16134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: Bootstrap starting.
I20260926 07:30:25.557420 16134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.558482 16134 log.cc:824] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.562551 16134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: No bootstrap required, opened a new log
I20260926 07:30:25.570554 16134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.570904 16134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.571060 16134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dba1c2d451498d88cf5b8934fa6f21, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.571748 16134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.573865 16309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.574277 16309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.575132 16134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:30:25.585295 16320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:25.585526 16320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:25.584472 16067 master_runner.cc:186] Error getting master registration for 127.2.147.253:33997: OK, instance_id { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" instance_seqno: 1790407824899866 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:30:25.595654 16067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:30:25.628734 16088 tablet.cc:2369] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:25.630847 16208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } attempt: 5
W20260926 07:30:25.638526 16236 tablet.cc:2369] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:30:25 dist-test-slave-1dwd krb5kdc[16054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:30:25.645937 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.631536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:44005 (local address 127.0.0.1:44018)
0926 07:30:25.631759 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.631779 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.631893 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.632280 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.632285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.632857 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.633129 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.633147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.634477 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.634483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.635471 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.635484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.636364 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.636369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.636609 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.637233 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.637251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.639656 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.642932 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.642938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.642942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.643535 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.643966 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.643970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.643974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.644139 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.644864 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.644872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.645041 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.645184 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.645522 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 07:30:25.647296 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.631574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.2.147.252:44005)
0926 07:30:25.631799 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.631807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.631826 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:30:25.632071 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.632082 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.632147 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.632248 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.633351 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.634323 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.635683 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.636208 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.636353 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.639857 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.639865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.639868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.639914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.642730 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.643674 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.643679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.643682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.643832 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.644269 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.644274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.644277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.644666 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.644804 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.646471 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.646749 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 07:30:25.651662 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: Bootstrap starting.
I20260926 07:30:25.655176 16208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.656284 16208 log.cc:824] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.658938 16208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: No bootstrap required, opened a new log
I20260926 07:30:25.668264 16208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.668635 16208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.668823 16208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7194d9c14bdd4b02b41d1f0cf6e1ad75, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.669545 16208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.670681 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.671265 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.672004 16208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:25.701059 16141 master_runner.cc:263] No leader master found from master 7194d9c14bdd4b02b41d1f0cf6e1ad75
I20260926 07:30:25.701249 16141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:30:25.704262 16334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:25.704514 16334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:25.758546 16309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:25.758865 16309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.761145 16309 leader_election.cc:288] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7194d9c14bdd4b02b41d1f0cf6e1ad75 (127.2.147.253:33997), 2d6d43970c8a4015b871b3cb7571a101 (127.2.147.252:44005)
I20260926 07:30:25.762145 16183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01dba1c2d451498d88cf5b8934fa6f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" is_pre_election: true
I20260926 07:30:25.762749 16183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dba1c2d451498d88cf5b8934fa6f21 in term 0.
I20260926 07:30:25.763556 16085 leader_election.cc:302] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01dba1c2d451498d88cf5b8934fa6f21, 7194d9c14bdd4b02b41d1f0cf6e1ad75; no voters:
I20260926 07:30:25.764765 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01dba1c2d451498d88cf5b8934fa6f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6d43970c8a4015b871b3cb7571a101" is_pre_election: true
I20260926 07:30:25.765440 16257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dba1c2d451498d88cf5b8934fa6f21 in term 0.
I20260926 07:30:25.766741 16309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:25.766918 16309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:25.767027 16309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.770699 16309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.772003 16183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01dba1c2d451498d88cf5b8934fa6f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75"
I20260926 07:30:25.772241 16183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.776295 16183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dba1c2d451498d88cf5b8934fa6f21 in term 1.
I20260926 07:30:25.776901 16085 leader_election.cc:302] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01dba1c2d451498d88cf5b8934fa6f21, 7194d9c14bdd4b02b41d1f0cf6e1ad75; no voters:
I20260926 07:30:25.780581 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01dba1c2d451498d88cf5b8934fa6f21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d6d43970c8a4015b871b3cb7571a101"
I20260926 07:30:25.780828 16257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.784403 16257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dba1c2d451498d88cf5b8934fa6f21 in term 1.
I20260926 07:30:25.785231 16309 leader_election.cc:288] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [CANDIDATE]: Term 1 election: Requested vote from peers 7194d9c14bdd4b02b41d1f0cf6e1ad75 (127.2.147.253:33997), 2d6d43970c8a4015b871b3cb7571a101 (127.2.147.252:44005)
I20260926 07:30:25.785868 16309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:25.786160 16309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [term 1 LEADER]: Becoming Leader. State: Replica: 01dba1c2d451498d88cf5b8934fa6f21, State: Running, Role: LEADER
I20260926 07:30:25.787386 16309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } }
I20260926 07:30:25.805769 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01dba1c2d451498d88cf5b8934fa6f21. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.806357 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:25.813583 16338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:25.816092 16338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:25.838546 16257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [term 1 FOLLOWER]: Refusing update from remote peer 01dba1c2d451498d88cf5b8934fa6f21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.839892 16336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.847608 16183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [term 1 FOLLOWER]: Refusing update from remote peer 01dba1c2d451498d88cf5b8934fa6f21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.848752 16336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.873710 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01dba1c2d451498d88cf5b8934fa6f21. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.874128 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.883675 16339 mvcc.cc:205] Tried to move back new op lower bound from 7333510454620151808 to 7333510454451458048. Current Snapshot: MvccSnapshot[applied={T|T < 7333510454620151808}]
I20260926 07:30:25.894090 16309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.894460 16309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:25.896291 16338 catalog_manager.cc:1347] Generated new cluster ID: 7f3e46f9dc3b44d9a94934485915ccbd
I20260926 07:30:25.896394 16338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:25.899708 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.900000 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.916869 16338 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:25.919480 16338 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:25.924935 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01dba1c2d451498d88cf5b8934fa6f21. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.925206 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:25.926911 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.927160 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:25.935549 16338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01dba1c2d451498d88cf5b8934fa6f21: Generated new TSK 0
I20260926 07:30:25.936707 16338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:25.947902 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01dba1c2d451498d88cf5b8934fa6f21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01dba1c2d451498d88cf5b8934fa6f21" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38909 } } peers { permanent_uuid: "7194d9c14bdd4b02b41d1f0cf6e1ad75" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 33997 } } peers { permanent_uuid: "2d6d43970c8a4015b871b3cb7571a101" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 44005 } } }
I20260926 07:30:25.948231 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:26.532855 16306 catalog_manager.cc:1259] Loaded cluster ID: 7f3e46f9dc3b44d9a94934485915ccbd
I20260926 07:30:26.532960 16306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: loading cluster ID for follower catalog manager: success
I20260926 07:30:26.535535 16306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: acquiring CA information for follower catalog manager: success
I20260926 07:30:26.537329 16306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d6d43970c8a4015b871b3cb7571a101: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:30:26.706580 16334 catalog_manager.cc:1259] Loaded cluster ID: 7f3e46f9dc3b44d9a94934485915ccbd
I20260926 07:30:26.706689 16334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: loading cluster ID for follower catalog manager: success
I20260926 07:30:26.709280 16334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: acquiring CA information for follower catalog manager: success
I20260926 07:30:26.711129 16334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7194d9c14bdd4b02b41d1f0cf6e1ad75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:30:27.081563 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16067
I20260926 07:30:27.082919 16129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:27.198480 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16067
I20260926 07:30:27.219655 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16141
I20260926 07:30:27.220758 16203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:27.327731 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16141
I20260926 07:30:27.348061 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16215
I20260926 07:30:27.349174 16277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:27.461114 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16215
2026-09-26T07:30:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:27 dist-test-slave-1dwd krb5kdc[16357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:27 dist-test-slave-1dwd krb5kdc[16357](info): set up 2 sockets
Sep 26 07:30:27 dist-test-slave-1dwd krb5kdc[16357](info): commencing operation
krb5kdc: starting...
W20260926 07:30:28.551833 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:30:28 dist-test-slave-1dwd krb5kdc[16357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:30:28.786806 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40579
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:40579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:29.124439 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:29.124747 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:29.124913 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:29.132470 16370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:29.132587 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:29.132681 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:29.132786 16370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:29.132853 16370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:29.143030 16370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:40579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:40579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:29.144518 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:29.145943 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:29.158313 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.158375 16375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.158862 16378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.159286 16370 server_base.cc:1034] running on GCE node
I20260926 07:30:29.160145 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:29.161693 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:29.162953 16370 hybrid_clock.cc:648] HybridClock initialized: now 1790407829162901 us; error 67 us; skew 500 ppm
I20260926 07:30:29.163403 16370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:29 dist-test-slave-1dwd krb5kdc[16357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:29.166683 16370 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:29.168375 16370 webserver.cc:460] Webserver started at http://127.2.147.254:45273/ using document root <none> and password file <none>
I20260926 07:30:29.169106 16370 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:29.169256 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:29.169584 16370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:29.172719 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "8e058aa2be714cb587a1eda168584054"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-1dwd"
server_key: "2a09000ffd15119e895230e365800121"
server_key_iv: "d830764dd206878a2c39072d79f84d67"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.173480 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "8e058aa2be714cb587a1eda168584054"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-1dwd"
server_key: "2a09000ffd15119e895230e365800121"
server_key_iv: "d830764dd206878a2c39072d79f84d67"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.178788 16370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:30:29.182315 16385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.183804 16370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:29.184020 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "8e058aa2be714cb587a1eda168584054"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-1dwd"
server_key: "2a09000ffd15119e895230e365800121"
server_key_iv: "d830764dd206878a2c39072d79f84d67"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.184245 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:29.201867 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:29.206799 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:29.207109 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:29.225271 16370 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:40579
I20260926 07:30:29.225291 16436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:40579 every 8 connection(s)
I20260926 07:30:29.226673 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:29.229707 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16370
I20260926 07:30:29.229960 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:30:29.230460 16437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:29.230679 2639 external_mini_cluster.cc:1428] Setting key 00232a25d73f3bb4a3781ac94faa2b0b
I20260926 07:30:29.245059 16437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054: Bootstrap starting.
Sep 26 07:30:29 dist-test-slave-1dwd krb5kdc[16357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:29.248950 16437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:29.250098 16437 log.cc:824] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:29.253106 16437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054: No bootstrap required, opened a new log
I20260926 07:30:29.255517 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.232557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.2.147.254:40579)
0926 07:30:29.233402 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:29.233418 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:29.233474 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:30:29.235649 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:29.235687 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:29.236219 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:29.236710 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:29.238038 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.239992 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.241356 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.242496 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.242692 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.246983 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:29.246995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:29.247021 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:29.247081 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:29.250704 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.251615 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.251622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.251628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.251801 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.252210 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.252216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.252219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.252771 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:29.253105 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:29.253588 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 07:30:29.254097 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":321,"threads_started":1}
I20260926 07:30:29.259773 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:29.262602 16437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } }
I20260926 07:30:29.263067 16437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:29.263172 16437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e058aa2be714cb587a1eda168584054, State: Initialized, Role: FOLLOWER
I20260926 07:30:29.263830 16437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } }
I20260926 07:30:29.264062 16437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:29.264202 16437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:29.264377 16437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:29.266512 16437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } }
I20260926 07:30:29.267004 16437 leader_election.cc:302] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e058aa2be714cb587a1eda168584054; no voters:
I20260926 07:30:29.267616 16437 leader_election.cc:288] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:29.267800 16442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:29.268524 16442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [term 1 LEADER]: Becoming Leader. State: Replica: 8e058aa2be714cb587a1eda168584054, State: Running, Role: LEADER
I20260926 07:30:29.269048 16442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } }
I20260926 07:30:29.269686 16437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:29.272198 16443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e058aa2be714cb587a1eda168584054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } } }
I20260926 07:30:29.272276 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e058aa2be714cb587a1eda168584054. Latest consensus state: current_term: 1 leader_uuid: "8e058aa2be714cb587a1eda168584054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e058aa2be714cb587a1eda168584054" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 40579 } } }
I20260926 07:30:29.272557 16443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:29.272653 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:29.275111 16449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:29.279742 16449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:29.287731 16449 catalog_manager.cc:1347] Generated new cluster ID: 09790c2577b24147bef3642594ce85d3
I20260926 07:30:29.287848 16449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:29.315892 16449 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:29.317255 16449 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:29.324450 16449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e058aa2be714cb587a1eda168584054: Generated new TSK 0
I20260926 07:30:29.325294 16449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:29.371773 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16370
I20260926 07:30:29.372943 16432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:29.470290 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16370
2026-09-26T07:30:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2004 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:29 dist-test-slave-1dwd krb5kdc[16465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:29 dist-test-slave-1dwd krb5kdc[16465](info): set up 2 sockets
Sep 26 07:30:29 dist-test-slave-1dwd krb5kdc[16465](info): commencing operation
krb5kdc: starting...
W20260926 07:30:30.575743 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:30:30 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:30:30.809329 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:31.141939 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:31.142261 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:31.142457 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:31.149957 16478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:31.150081 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:31.150203 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:31.150281 16478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:31.150321 16478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:31.160501 16478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:31.161923 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:31.163281 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:31.175405 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:31.175503 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:31.175436 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:31.180717 16478 server_base.cc:1034] running on GCE node
I20260926 07:30:31.181586 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:31.183084 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:31.184295 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790407831184247 us; error 36 us; skew 500 ppm
I20260926 07:30:31.184664 16478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:31 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407831, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:31.187990 16478 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:31.189515 16478 webserver.cc:460] Webserver started at http://127.2.147.254:38065/ using document root <none> and password file <none>
I20260926 07:30:31.190255 16478 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:31.190402 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:31.190712 16478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:31.193943 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "f71a8bce1c2749de9f5a55e988760640"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "a50506db129bdb0915ae90664b4b79bb"
server_key_iv: "fd6348a40d2e4228e6fe19bfb108d573"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.194758 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "f71a8bce1c2749de9f5a55e988760640"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "a50506db129bdb0915ae90664b4b79bb"
server_key_iv: "fd6348a40d2e4228e6fe19bfb108d573"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.200304 16478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 07:30:31.203786 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.205168 16478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:31.205356 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f71a8bce1c2749de9f5a55e988760640"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "a50506db129bdb0915ae90664b4b79bb"
server_key_iv: "fd6348a40d2e4228e6fe19bfb108d573"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.205559 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:31.231343 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:31.235893 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:31.236210 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:31.256281 16478 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41439
I20260926 07:30:31.256312 16544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41439 every 8 connection(s)
I20260926 07:30:31.257745 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:31.261565 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:31.263073 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16478
I20260926 07:30:31.263317 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:30:31.264181 2639 external_mini_cluster.cc:1428] Setting key 8f2f2cf138b1f1233f84ba4c61615391
I20260926 07:30:31.276712 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap starting.
Sep 26 07:30:31 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:31.280728 16545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:31.281965 16545 log.cc:824] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:31.285148 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: No bootstrap required, opened a new log
I20260926 07:30:31.288632 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.266318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.2.147.254:41439)
0926 07:30:31.267254 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.267279 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.267324 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:30:31.268900 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.268924 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.269346 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.269691 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.271535 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.273081 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.274983 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.275759 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.275892 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.280831 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:31.280840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:31.280879 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:31.280938 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:31.284410 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.285164 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.285171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.285179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.285387 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.285772 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.285778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.285781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.286307 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:31.286633 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:31.287114 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 07:30:31.287577 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":425,"threads_started":1}
I20260926 07:30:31.294019 16545 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:31.294469 16545 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:31.294588 16545 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Initialized, Role: FOLLOWER
I20260926 07:30:31.295325 16545 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:31.295617 16545 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:31.295770 16545 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:31.295940 16545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:31.298518 16545 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:31.299065 16545 leader_election.cc:302] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71a8bce1c2749de9f5a55e988760640; no voters:
I20260926 07:30:31.299707 16545 leader_election.cc:288] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:31.299868 16550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:31.300556 16550 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 LEADER]: Becoming Leader. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Running, Role: LEADER
I20260926 07:30:31.301091 16550 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:31.301776 16545 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:31.304224 16552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71a8bce1c2749de9f5a55e988760640. Latest consensus state: current_term: 1 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:31.304579 16552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:31.304345 16551 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:31.304806 16551 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:31.307458 16558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:31.312513 16558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:31.321792 16558 catalog_manager.cc:1347] Generated new cluster ID: a4070613dbdf441dbb59a3abadd2b217
I20260926 07:30:31.321918 16558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:31.374986 16558 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:31.376927 16558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:31.388375 16558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Generated new TSK 0
I20260926 07:30:31.389456 16558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:30:31.442992 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:31.774760 16571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:31.775027 16571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:31.775259 16571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:31.783195 16571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:31.783372 16571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:31.783485 16571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:31.793577 16571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:31.794952 16571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:31.796411 16571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:31.810178 16576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:31.810092 16579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:31.810505 16577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:31.810947 16571 server_base.cc:1034] running on GCE node
I20260926 07:30:31.811635 16571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:31.812595 16571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:31.813925 16571 hybrid_clock.cc:648] HybridClock initialized: now 1790407831813878 us; error 54 us; skew 500 ppm
I20260926 07:30:31.814491 16571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:31 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407831, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:31.817750 16571 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:31.819258 16571 webserver.cc:460] Webserver started at http://127.2.147.193:34843/ using document root <none> and password file <none>
I20260926 07:30:31.819972 16571 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:31.820120 16571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:31.820423 16571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:31.823372 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "02f43d1dbace4808a6a07c46a8430467"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "18f1c4b4744ce3c9c34cb79669bdee55"
server_key_iv: "a19d64c1436a95634f453288efa8f646"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.824126 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "02f43d1dbace4808a6a07c46a8430467"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "18f1c4b4744ce3c9c34cb79669bdee55"
server_key_iv: "a19d64c1436a95634f453288efa8f646"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.829221 16571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:30:31.832624 16586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.833874 16571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:31.834055 16571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "02f43d1dbace4808a6a07c46a8430467"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "18f1c4b4744ce3c9c34cb79669bdee55"
server_key_iv: "a19d64c1436a95634f453288efa8f646"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.834259 16571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:31.862192 16571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:31.866369 16571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:31.866686 16571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:31.867730 16571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:31.869072 16571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:31.869190 16571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.869346 16571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:31.869419 16571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.902407 16571 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38201
I20260926 07:30:31.902448 16698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38201 every 8 connection(s)
I20260926 07:30:31.903860 16571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:31.906100 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16571
I20260926 07:30:31.906324 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:31.907234 2639 external_mini_cluster.cc:1428] Setting key 32dbee9e5e66c9e3e9669dbc4397c47f
Sep 26 07:30:31 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407831, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:31.924476 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.906469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43297 (local address 127.2.147.254:41439)
0926 07:30:31.907013 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.907021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.908197 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 07:30:31.910256 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.910264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.910351 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.910468 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.911969 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.913146 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.914688 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.915417 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.915564 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.918906 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:31.918911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:31.918914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:31.918945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:31.921480 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.922206 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.922213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.922217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.922340 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.922758 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.922765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.922768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.923027 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:31.923198 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:31.923883 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 07:30:31.924115 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260926 07:30:31.925544 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.907181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:43297)
0926 07:30:31.908032 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.908078 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.909914 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.910633 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.910643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.911122 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.911788 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.911806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.913308 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.913312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.914527 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.914538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.915618 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.915623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.915819 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.916559 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.916582 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.918690 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:31.921635 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:31.921644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:31.921659 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:31.922079 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:31.922451 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:31.922456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:31.922460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:31.922643 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:31.923154 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:31.923162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:31.923530 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 07:30:31.923792 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 07:30:31.924314 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260926 07:30:31.927237 16699 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:31.927552 16699 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:31.928270 16699 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:31.930446 16510 ts_manager.cc:194] Registered new tserver with Master: 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
I20260926 07:30:31.932189 16510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:43297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:30:31.958572 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:32.283589 16705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:32.283877 16705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:32.284040 16705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:32.291352 16705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:32.291462 16705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:32.291666 16705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:32.301494 16705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:32.302841 16705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:32.304129 16705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:32.317129 16711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:32.317215 16710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:32.317189 16713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:32.317975 16705 server_base.cc:1034] running on GCE node
I20260926 07:30:32.318620 16705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:32.319609 16705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:32.320812 16705 hybrid_clock.cc:648] HybridClock initialized: now 1790407832320767 us; error 35 us; skew 500 ppm
I20260926 07:30:32.321131 16705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:32 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:32.324358 16705 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:32.325846 16705 webserver.cc:460] Webserver started at http://127.2.147.194:35283/ using document root <none> and password file <none>
I20260926 07:30:32.326558 16705 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:32.326709 16705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:32.327016 16705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:32.330036 16705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "04983ce7c07122d9461d7f7a7260cec3"
server_key_iv: "483380d0778e45807204f4f61d448778"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.330899 16705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "04983ce7c07122d9461d7f7a7260cec3"
server_key_iv: "483380d0778e45807204f4f61d448778"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.336206 16705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:30:32.339790 16720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.341202 16705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:32.341378 16705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "04983ce7c07122d9461d7f7a7260cec3"
server_key_iv: "483380d0778e45807204f4f61d448778"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.341579 16705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:32.373857 16705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:32.378420 16705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:32.378732 16705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:32.379782 16705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:32.381206 16705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:32.381306 16705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.381457 16705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:32.381529 16705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.419013 16705 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37233
I20260926 07:30:32.419054 16832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37233 every 8 connection(s)
I20260926 07:30:32.420357 16705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:32.422043 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16705
I20260926 07:30:32.422243 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:32.422802 2639 external_mini_cluster.cc:1428] Setting key 2eb216cdea5b08f36c375550584ae4e9
Sep 26 07:30:32 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:32.440634 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.423167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:56387 (local address 127.2.147.254:41439)
0926 07:30:32.423372 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.423376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.424821 (+ 1445us) server_negotiation.cc:407] Connection header received
0926 07:30:32.426656 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.426662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.426755 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.426863 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.428385 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.429100 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.430656 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.431061 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.431234 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.434675 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:32.434680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:32.434684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:32.434717 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:32.437404 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.438195 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.438201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.438210 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.438333 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.438855 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.438858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.438860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.439151 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:32.439351 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:32.440046 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 07:30:32.440289 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:30:32.441612 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.423807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:56387)
0926 07:30:32.424603 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:32.424689 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:32.426388 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:32.427102 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:32.427110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:32.427652 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:32.428216 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.428233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.429270 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.429274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.430500 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.430512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.431316 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.431325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.431562 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.432341 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:32.432366 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:32.434449 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:32.437556 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:32.437565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:32.437581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:32.438056 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:32.438448 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:32.438453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:32.438457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:32.438727 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:32.439312 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:32.439320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:32.439686 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 07:30:32.439950 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 07:30:32.440479 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260926 07:30:32.443082 16833 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:32.443420 16833 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:32.444116 16833 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
I20260926 07:30:32.445683 16510 ts_manager.cc:194] Registered new tserver with Master: cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
I20260926 07:30:32.446585 16510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:56387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:30:32.472465 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:32.795471 16838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:32.795749 16838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:32.795935 16838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:32.803285 16838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:32.803392 16838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:32.803589 16838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:32.813617 16838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:32.814945 16838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:32.816361 16838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:32.829260 16843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:32.829286 16846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:32.829308 16844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:32.830077 16838 server_base.cc:1034] running on GCE node
I20260926 07:30:32.830715 16838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:32.831753 16838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:32.832980 16838 hybrid_clock.cc:648] HybridClock initialized: now 1790407832832938 us; error 66 us; skew 500 ppm
I20260926 07:30:32.833299 16838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:32 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:32.836416 16838 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:32.837860 16838 webserver.cc:460] Webserver started at http://127.2.147.195:32985/ using document root <none> and password file <none>
I20260926 07:30:32.838562 16838 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:32.838706 16838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:32.839004 16838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:32.841925 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "1ae74f082b7e440babd7163a4ec55357"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "ac3d6a3e9ea844daa95d5f2ad9a6c14a"
server_key_iv: "49b83f6faa1b42eeb9f9933701b94706"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.842695 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "1ae74f082b7e440babd7163a4ec55357"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "ac3d6a3e9ea844daa95d5f2ad9a6c14a"
server_key_iv: "49b83f6faa1b42eeb9f9933701b94706"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.848476 16838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 07:30:32.852039 16853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.853338 16838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:32.853514 16838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "1ae74f082b7e440babd7163a4ec55357"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "ac3d6a3e9ea844daa95d5f2ad9a6c14a"
server_key_iv: "49b83f6faa1b42eeb9f9933701b94706"
server_key_version: "encryptionkey@0"
I20260926 07:30:32.853730 16838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:32.885746 16838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:32.889931 16838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:32.890259 16838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:32.891322 16838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:32.892735 16838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:32.892845 16838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.893009 16838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:32.893090 16838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:32.927515 16838 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:45773
I20260926 07:30:32.927562 16965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:45773 every 8 connection(s)
I20260926 07:30:32.928830 16838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:32.934876 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 16838
I20260926 07:30:32.935081 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:32.935302 16699 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:32.935658 2639 external_mini_cluster.cc:1428] Setting key 86174014b4826ef083777500f38ceb60
Sep 26 07:30:32 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:32.949079 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.931427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:57133 (local address 127.2.147.254:41439)
0926 07:30:32.931624 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.931628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.932783 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 07:30:32.934577 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.934581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.934651 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.934737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.936878 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.937906 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.939839 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.940246 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.940344 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.943922 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:32.943927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:32.943930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:32.943962 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:32.946413 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.947092 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.947096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.947098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.947271 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.947648 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.947651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.947653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.947920 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:32.948033 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:32.948663 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 07:30:32.948836 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:30:32.950300 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.931933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:57133)
0926 07:30:32.932609 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:32.932652 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:32.934267 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:32.935045 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:32.935059 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:32.935830 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:32.936646 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.936680 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.938050 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.938054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.939714 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.939725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.940536 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.940541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.940753 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.941359 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:32.941384 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:32.943729 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:32.946538 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:32.946544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:32.946573 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:32.946991 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:32.947372 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:32.947376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:32.947378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:32.947548 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:32.948024 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:32.948030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:32.948340 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:30:32.948560 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 07:30:32.948944 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":276,"threads_started":1}
I20260926 07:30:32.951659 16966 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:32.951972 16966 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:32.952675 16966 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
I20260926 07:30:32.953985 16509 ts_manager.cc:194] Registered new tserver with Master: 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773)
I20260926 07:30:32.954866 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:57133
I20260926 07:30:32.963402 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:32.979609 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.967881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.2.147.254:41439)
0926 07:30:32.968045 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.968049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.968410 (+ 361us) server_negotiation.cc:407] Connection header received
0926 07:30:32.968541 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.968544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.968615 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.968707 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.969978 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.970701 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.971993 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.972403 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.972496 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.974826 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:32.974831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:32.974834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:32.974862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:32.977419 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.977998 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.978001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.978003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.978124 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:32.978426 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:32.978429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:32.978431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:32.978739 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:32.978912 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:32.979169 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:30:32.979393 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:30:32.983760 16509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:32.990468 16509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:33.015080 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.003494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.2.147.193:38201)
0926 07:30:33.004129 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.004138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.004166 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:30:33.007526 (+ 3360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.007536 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.007845 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.008658 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:33.008885 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.010318 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.012419 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.013868 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.014015 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.014252 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.014529 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.014730 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":298,"threads_started":1}
I20260926 07:30:33.015164 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.003225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57772)
0926 07:30:33.003998 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.004043 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.004228 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.008035 (+ 3807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.008040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.008093 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:33.008546 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.008555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.010445 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.010451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.012256 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.012268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.014056 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.014079 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.014268 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.014321 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:30:33.014468 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:30:33.014678 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":56,"spinlock_wait_cycles":2304,"thread_start_us":333,"threads_started":1}
I20260926 07:30:33.015614 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.004332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37233 (local address 127.0.0.1:53890)
0926 07:30:33.004877 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.004893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.005001 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.007993 (+ 2992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.007999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.008029 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:33.008532 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.008545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.010380 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.010386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.012869 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.012883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.014710 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.014717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.014891 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.014935 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:30:33.015005 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:30:33.015158 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":89,"thread_start_us":247,"threads_started":1}
I20260926 07:30:33.015998 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.005782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:45773 (local address 127.0.0.1:49792)
0926 07:30:33.006258 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.006270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.006406 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.007278 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.007288 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.007319 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:33.007655 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.007668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.010021 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.010027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.012813 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.012824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.014934 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.014939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.015141 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.015182 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:30:33.015301 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:30:33.015437 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":58,"thread_start_us":335,"threads_started":1}
I20260926 07:30:33.016238 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.005353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.2.147.195:45773)
0926 07:30:33.006048 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.006059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.006363 (+ 304us) server_negotiation.cc:407] Connection header received
0926 07:30:33.006542 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.006549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.006825 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.007020 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:33.007913 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.009867 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.013032 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.014472 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.014581 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.014794 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.015696 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.015865 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":543,"threads_started":1}
I20260926 07:30:33.016578 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.004677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.2.147.194:37233)
0926 07:30:33.007512 (+ 2835us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.007520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.007547 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:30:33.007626 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.007632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.007809 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.008016 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:33.008766 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.010230 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.013045 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.014477 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.015854 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.016036 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.016095 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.016259 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2682,"thread_start_us":2531,"threads_started":1}
I20260926 07:30:33.019529 16768 tablet_service.cc:1467] Processing CreateTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 (DEFAULT_TABLE table=test-table [id=000b5983733a461f8c9825564f2c1fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:33.019543 16634 tablet_service.cc:1467] Processing CreateTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 (DEFAULT_TABLE table=test-table [id=000b5983733a461f8c9825564f2c1fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:33.020900 16634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75bbb27d484ce8afb56be4fb2e96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:33.020896 16768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75bbb27d484ce8afb56be4fb2e96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:33.020606 16901 tablet_service.cc:1467] Processing CreateTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 (DEFAULT_TABLE table=test-table [id=000b5983733a461f8c9825564f2c1fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:33.021878 16901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75bbb27d484ce8afb56be4fb2e96b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:33.032706 16984 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Bootstrap starting.
I20260926 07:30:33.035337 16985 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Bootstrap starting.
I20260926 07:30:33.036304 16984 tablet_bootstrap.cc:654] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:33.037943 16984 log.cc:824] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:33.039458 16985 tablet_bootstrap.cc:654] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:33.040639 16986 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Bootstrap starting.
I20260926 07:30:33.040814 16985 log.cc:824] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:33.041783 16984 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: No bootstrap required, opened a new log
I20260926 07:30:33.042066 16984 ts_tablet_manager.cc:1397] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 07:30:33.043744 16986 tablet_bootstrap.cc:654] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:33.043864 16985 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: No bootstrap required, opened a new log
I20260926 07:30:33.044135 16985 ts_tablet_manager.cc:1397] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 07:30:33.044813 16986 log.cc:824] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:33.047827 16986 tablet_bootstrap.cc:492] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: No bootstrap required, opened a new log
I20260926 07:30:33.048105 16986 ts_tablet_manager.cc:1397] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:30:33.053366 16984 raft_consensus.cc:348] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.053833 16984 raft_consensus.cc:374] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:33.053974 16984 raft_consensus.cc:729] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f43d1dbace4808a6a07c46a8430467, State: Initialized, Role: FOLLOWER
I20260926 07:30:33.054409 16985 raft_consensus.cc:348] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.054814 16985 raft_consensus.cc:374] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:33.054630 16984 consensus_queue.cc:260] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.054911 16985 raft_consensus.cc:729] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5aa0ded97c4b6681bcdd4374d65ae0, State: Initialized, Role: FOLLOWER
I20260926 07:30:33.055127 16986 raft_consensus.cc:348] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.055624 16986 raft_consensus.cc:374] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:33.055737 16986 raft_consensus.cc:729] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae74f082b7e440babd7163a4ec55357, State: Initialized, Role: FOLLOWER
I20260926 07:30:33.056259 16986 consensus_queue.cc:260] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.056440 16984 ts_tablet_manager.cc:1428] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 07:30:33.055589 16985 consensus_queue.cc:260] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.057575 16966 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:33.057945 16986 ts_tablet_manager.cc:1428] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 07:30:33.059882 16833 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:33.060478 16985 ts_tablet_manager.cc:1428] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260926 07:30:33.156471 16700 tablet.cc:2369] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:30:33.172376 16834 tablet.cc:2369] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:30:33.180454 16967 tablet.cc:2369] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:33.324388 16991 raft_consensus.cc:482] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:33.324658 16991 raft_consensus.cc:504] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.326213 16991 leader_election.cc:288] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
I20260926 07:30:33.331875 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.326707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:48195 (local address 127.2.147.193:38201)
0926 07:30:33.326885 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.326889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.326909 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:33.327168 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.327173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.327277 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.327398 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:33.327793 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.328526 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.330125 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.330876 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.331036 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.331109 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.331494 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.331617 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:33.331876 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.326596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.2.147.195:48195)
0926 07:30:33.326811 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.326832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.326963 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.327415 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.327420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.327444 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:33.327649 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.327657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.328677 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.328682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.329957 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.329968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.331031 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.331037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.331338 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.331364 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:30:33.331467 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:30:33.331598 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":42}
I20260926 07:30:33.332984 16654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75bbb27d484ce8afb56be4fb2e96b8" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f43d1dbace4808a6a07c46a8430467" is_pre_election: true
I20260926 07:30:33.333251 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.327191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:33359 (local address 127.2.147.194:37233)
0926 07:30:33.327467 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.327476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.328078 (+ 602us) server_negotiation.cc:407] Connection header received
0926 07:30:33.328269 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.328274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.328330 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.328424 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:33.328975 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.329685 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.331389 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.332368 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.332533 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.332600 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.332869 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.332993 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:33.333328 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.327017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37233 (local address 127.2.147.195:33359)
0926 07:30:33.327971 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.327984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.328120 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.328437 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.328441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.328462 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:33.328818 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.328826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.329843 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.329848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.331181 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.331192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.332545 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.332550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.332714 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.332737 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:30:33.332841 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:30:33.332961 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":24,"thread_start_us":664,"threads_started":1}
I20260926 07:30:33.333561 16654 raft_consensus.cc:2454] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 0.
I20260926 07:30:33.334204 16788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75bbb27d484ce8afb56be4fb2e96b8" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" is_pre_election: true
I20260926 07:30:33.334293 16854 leader_election.cc:302] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467, 1ae74f082b7e440babd7163a4ec55357; no voters:
I20260926 07:30:33.334702 16991 raft_consensus.cc:2781] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:33.334702 16788 raft_consensus.cc:2454] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 0.
I20260926 07:30:33.334863 16991 raft_consensus.cc:482] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:33.334973 16991 raft_consensus.cc:3037] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:33.337307 16991 raft_consensus.cc:504] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.337950 16991 leader_election.cc:288] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 election: Requested vote from peers 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
I20260926 07:30:33.338368 16654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75bbb27d484ce8afb56be4fb2e96b8" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f43d1dbace4808a6a07c46a8430467"
I20260926 07:30:33.338424 16788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75bbb27d484ce8afb56be4fb2e96b8" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
I20260926 07:30:33.338603 16654 raft_consensus.cc:3037] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:33.338605 16788 raft_consensus.cc:3037] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:33.341856 16788 raft_consensus.cc:2454] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 1.
I20260926 07:30:33.341856 16654 raft_consensus.cc:2454] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 1.
I20260926 07:30:33.342250 16854 leader_election.cc:302] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467, 1ae74f082b7e440babd7163a4ec55357; no voters:
I20260926 07:30:33.342540 16991 raft_consensus.cc:2781] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:33.342995 16991 raft_consensus.cc:686] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae74f082b7e440babd7163a4ec55357, State: Running, Role: LEADER
I20260926 07:30:33.343515 16991 consensus_queue.cc:237] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } }
I20260926 07:30:33.347906 16509 catalog_manager.cc:5495] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "1ae74f082b7e440babd7163a4ec55357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } health_report { overall_health: HEALTHY } } }
I20260926 07:30:33.385046 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.379909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.2.147.195:45773)
0926 07:30:33.380142 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.380148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.380286 (+ 138us) server_negotiation.cc:407] Connection header received
0926 07:30:33.380454 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.380458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.380548 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.380685 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:33.381011 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.381700 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.383038 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.383493 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.383586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.384055 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:33.384543 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:33.384665 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.384784 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.384856 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:30:33.393036 16788 raft_consensus.cc:1261] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Refusing update from remote peer 1ae74f082b7e440babd7163a4ec55357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:33.393100 16654 raft_consensus.cc:1261] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Refusing update from remote peer 1ae74f082b7e440babd7163a4ec55357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:33.393996 16994 consensus_queue.cc:1035] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:33.394358 16991 consensus_queue.cc:1035] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:33.417416 16510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:33.417680 16510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:33.421711 16510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Sending DeleteTablet for 3 replicas of tablet fa75bbb27d484ce8afb56be4fb2e96b8
I20260926 07:30:33.422837 16634 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:33 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:57772
I20260926 07:30:33.423296 16768 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:33 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53890
I20260926 07:30:33.423969 16901 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75bbb27d484ce8afb56be4fb2e96b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:33 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:49792
I20260926 07:30:33.424415 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16571
I20260926 07:30:33.437304 17008 tablet_replica.cc:330] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: stopping tablet replica
I20260926 07:30:33.438416 17009 tablet_replica.cc:330] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: stopping tablet replica
I20260926 07:30:33.438922 17009 raft_consensus.cc:2229] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:33.439308 17008 raft_consensus.cc:2229] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:33.439749 17009 raft_consensus.cc:2258] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:33.439999 17008 raft_consensus.cc:2258] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:33.442162 17008 ts_tablet_manager.cc:1905] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:33.442202 17009 ts_tablet_manager.cc:1905] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:33.451223 17009 ts_tablet_manager.cc:1918] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:33.451190 17008 ts_tablet_manager.cc:1918] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:33.451417 17009 log.cc:1196] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/fa75bbb27d484ce8afb56be4fb2e96b8
I20260926 07:30:33.452091 17009 ts_tablet_manager.cc:1939] T fa75bbb27d484ce8afb56be4fb2e96b8 P 1ae74f082b7e440babd7163a4ec55357: Deleting consensus metadata
I20260926 07:30:33.453543 17007 tablet_replica.cc:330] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: stopping tablet replica
I20260926 07:30:33.454005 16494 catalog_manager.cc:4841] TS 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): tablet fa75bbb27d484ce8afb56be4fb2e96b8 (table test-table [id=000b5983733a461f8c9825564f2c1fae]) successfully deleted
I20260926 07:30:33.454584 17007 raft_consensus.cc:2229] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:33.455123 17007 raft_consensus.cc:2258] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:33.457476 17008 log.cc:1196] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/fa75bbb27d484ce8afb56be4fb2e96b8
I20260926 07:30:33.457489 17007 ts_tablet_manager.cc:1905] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:33.458105 17008 ts_tablet_manager.cc:1939] T fa75bbb27d484ce8afb56be4fb2e96b8 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting consensus metadata
I20260926 07:30:33.460047 16494 catalog_manager.cc:4841] TS cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): tablet fa75bbb27d484ce8afb56be4fb2e96b8 (table test-table [id=000b5983733a461f8c9825564f2c1fae]) successfully deleted
Sep 26 07:30:33 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:30:33.464519 17007 ts_tablet_manager.cc:1918] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:33.464715 17007 log.cc:1196] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/fa75bbb27d484ce8afb56be4fb2e96b8
I20260926 07:30:33.465322 17007 ts_tablet_manager.cc:1939] T fa75bbb27d484ce8afb56be4fb2e96b8 P 02f43d1dbace4808a6a07c46a8430467: Deleting consensus metadata
I20260926 07:30:33.467120 16494 catalog_manager.cc:4841] TS 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201): tablet fa75bbb27d484ce8afb56be4fb2e96b8 (table test-table [id=000b5983733a461f8c9825564f2c1fae]) successfully deleted
I20260926 07:30:33.470981 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.425554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.2.147.193:38201)
0926 07:30:33.425752 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.425758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.450341 (+ 24583us) server_negotiation.cc:407] Connection header received
0926 07:30:33.450836 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.450843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.450949 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.451038 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.454153 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.455363 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.457446 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.458076 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.458187 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.463769 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:33.463784 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:33.463802 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:33.463848 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:33.468147 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.468827 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.468837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.468840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.469002 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.469383 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.469389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.469392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.469861 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:33.470177 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.470370 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.470587 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:30:33.471753 16694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:33.563937 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16571
I20260926 07:30:33.582091 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16705
Sep 26 07:30:33 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:30:33.599254 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.582952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.2.147.194:37233)
0926 07:30:33.583117 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.583122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.583146 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:30:33.583356 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.583360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.583433 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.583519 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.585177 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.585867 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.587856 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.588241 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.588336 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.593370 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:33.593384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:33.593400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:33.593434 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:33.596337 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.597046 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.597053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.597056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.597197 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.597591 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.597595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.597597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.597968 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:33.598294 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.598615 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.598837 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:30:33.599906 16828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:33.717381 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16705
I20260926 07:30:33.734580 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16838
Sep 26 07:30:33 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:30:33.752971 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.735520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.2.147.195:45773)
0926 07:30:33.735702 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.735706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.735724 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:30:33.735918 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.735922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.735998 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.736082 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.737904 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.738762 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.740722 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.741182 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.741283 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.746673 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:33.746694 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:33.746711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:33.746752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:33.749923 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.750656 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.750664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.750666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.750838 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:33.751260 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:33.751267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:33.751275 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:33.751694 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:33.752028 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:33.752348 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 07:30:33.752571 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:30:33.753733 16961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:33.879388 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16838
I20260926 07:30:33.899471 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 16478
I20260926 07:30:33.900635 16540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:34.012895 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 16478
I20260926 07:30:34.035847 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--webserver_interface=127.2.147.254
--webserver_port=38065
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:34.362844 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:34.363145 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:34.363348 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:34.370673 17015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:34.370769 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:34.370867 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:34.370968 17015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:34.371034 17015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:34.381129 17015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=38065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:34.382421 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:34.383778 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:34.405791 17021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:34.405886 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:34.405920 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:34.406818 17015 server_base.cc:1034] running on GCE node
I20260926 07:30:34.407729 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:34.409404 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:34.410616 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790407834410575 us; error 30 us; skew 500 ppm
I20260926 07:30:34.411012 17015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:34 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:34.418430 17015 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:34.420255 17015 webserver.cc:460] Webserver started at http://127.2.147.254:38065/ using document root <none> and password file <none>
I20260926 07:30:34.421056 17015 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:34.421217 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:34.427405 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:34.430438 17031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:34.431939 17015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:34.432154 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f71a8bce1c2749de9f5a55e988760640"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "a50506db129bdb0915ae90664b4b79bb"
server_key_iv: "fd6348a40d2e4228e6fe19bfb108d573"
server_key_version: "encryptionkey@0"
I20260926 07:30:34.433292 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:34.450201 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:34.454977 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:34.455327 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:34.474299 17015 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41439
I20260926 07:30:34.474319 17082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41439 every 8 connection(s)
I20260926 07:30:34.475939 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:34.479127 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17015
I20260926 07:30:34.480237 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:38201
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=34843
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:34.485574 17083 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:34.490631 17083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap starting.
I20260926 07:30:34.511587 17083 log.cc:824] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:34.524077 17083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:34.524670 17083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap complete.
I20260926 07:30:34.537393 17083 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:34.538887 17083 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Initialized, Role: FOLLOWER
I20260926 07:30:34.539786 17083 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:34.540136 17083 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:34.540264 17083 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:34.540454 17083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:30:34.545527 17083 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:34.546243 17083 leader_election.cc:302] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71a8bce1c2749de9f5a55e988760640; no voters:
I20260926 07:30:34.546962 17083 leader_election.cc:288] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:30:34.547259 17087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:30:34.549573 17083 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:34.551832 17087 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 LEADER]: Becoming Leader. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Running, Role: LEADER
I20260926 07:30:34.552640 17087 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:34.555698 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:34.556048 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:34.559489 17089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71a8bce1c2749de9f5a55e988760640. Latest consensus state: current_term: 2 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:34.560022 17089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:34.567895 17097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:34.576226 17097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:34.577662 17097 catalog_manager.cc:1259] Loaded cluster ID: a4070613dbdf441dbb59a3abadd2b217
I20260926 07:30:34.577766 17097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:34.581677 17097 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:30:34.586450 17104 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:30:34.603896 17097 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:30:34.612457 17097 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:34.613788 17097 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Loaded TSK: 0
I20260926 07:30:34.614885 17097 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:30:34.873580 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:34.873865 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:34.874028 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:34.881585 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:34.881699 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:34.881875 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:34.891990 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:38201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=34843
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:34.893356 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:34.894707 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:34.910130 17111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:34.910665 17110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:34.911477 17113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:34.913156 17085 server_base.cc:1034] running on GCE node
I20260926 07:30:34.913830 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:34.914863 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:34.916102 17085 hybrid_clock.cc:648] HybridClock initialized: now 1790407834916027 us; error 73 us; skew 500 ppm
I20260926 07:30:34.916429 17085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:34 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:34.924055 17085 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:34.925774 17085 webserver.cc:460] Webserver started at http://127.2.147.193:34843/ using document root <none> and password file <none>
I20260926 07:30:34.926612 17085 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:34.926759 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:34.932663 17085 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 07:30:34.935822 17120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:34.937734 17085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 07:30:34.937966 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "02f43d1dbace4808a6a07c46a8430467"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "18f1c4b4744ce3c9c34cb79669bdee55"
server_key_iv: "a19d64c1436a95634f453288efa8f646"
server_key_version: "encryptionkey@0"
I20260926 07:30:34.939035 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:34.970168 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:34.975275 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:34.975630 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:34.976806 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:34.978317 17085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:34.978407 17085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:34.978567 17085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:34.978669 17085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.013870 17085 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38201
I20260926 07:30:35.013955 17232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38201 every 8 connection(s)
I20260926 07:30:35.015409 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:35.018503 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17085
I20260926 07:30:35.019261 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:37233
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=35283
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:35 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:35.045120 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.018513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:60113)
0926 07:30:35.023839 (+ 5326us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.023890 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.025678 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.027183 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.027195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.027846 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.028384 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.028401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.030114 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.030118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.031467 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.031480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.032527 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.032534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.032822 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.033538 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.033562 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.035801 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:35.040810 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.040818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.040831 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.041256 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.041806 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.041809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.041811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.041992 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.042941 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:35.042947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:35.043312 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 07:30:35.043607 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 07:30:35.044070 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5021,"thread_start_us":4783,"threads_started":1}
I20260926 07:30:35.045455 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.018348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60113 (local address 127.2.147.254:41439)
0926 07:30:35.019052 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.019073 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.024066 (+ 4993us) server_negotiation.cc:407] Connection header received
0926 07:30:35.026285 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.026299 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.026659 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.027034 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.028620 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.029932 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.031623 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.032282 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.032492 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.036092 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:35.036100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:35.036116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:35.036159 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:35.040642 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.041416 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.041429 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.041438 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.041691 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.042112 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.042118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.042121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.042760 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:35.042973 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:35.043871 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 07:30:35.044309 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":271,"threads_started":1}
I20260926 07:30:35.048280 17233 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:35.048585 17233 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:35.049304 17233 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
I20260926 07:30:35.051744 17048 ts_manager.cc:194] Registered new tserver with Master: 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201)
I20260926 07:30:35.053721 17048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:60113
W20260926 07:30:35.352893 17237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:35.353183 17237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:35.353277 17237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:35.361058 17237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:35.361167 17237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:35.361281 17237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:35.371181 17237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:37233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=35283
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:35.372502 17237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:35.373747 17237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:35.386932 17242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:35.386890 17245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:35.386981 17243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:35.392596 17237 server_base.cc:1034] running on GCE node
I20260926 07:30:35.393177 17237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:35.394042 17237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:35.395269 17237 hybrid_clock.cc:648] HybridClock initialized: now 1790407835395185 us; error 28 us; skew 500 ppm
I20260926 07:30:35.395592 17237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:35 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:35.398593 17237 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:35.400087 17237 webserver.cc:460] Webserver started at http://127.2.147.194:35283/ using document root <none> and password file <none>
I20260926 07:30:35.400813 17237 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:35.400970 17237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:35.406296 17237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:30:35.408984 17252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.410283 17237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:35.410462 17237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "04983ce7c07122d9461d7f7a7260cec3"
server_key_iv: "483380d0778e45807204f4f61d448778"
server_key_version: "encryptionkey@0"
I20260926 07:30:35.411507 17237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:35.425818 17237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:35.429641 17237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:35.429919 17237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:35.430881 17237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:35.432348 17237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:35.432469 17237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.432608 17237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:35.432699 17237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.465711 17237 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37233
I20260926 07:30:35.465802 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37233 every 8 connection(s)
I20260926 07:30:35.467242 17237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:35.472599 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17237
I20260926 07:30:35.473217 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:45773
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=32985
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:35 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:35.488727 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.469667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58543 (local address 127.2.147.254:41439)
0926 07:30:35.469856 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.469861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.471012 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 07:30:35.472920 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.472926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.473029 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.473141 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.474734 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.475523 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.477058 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.477526 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.477656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.481043 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:35.481047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:35.481050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:35.481082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:35.485010 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.485932 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.485940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.485947 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.486109 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.486588 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.486595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.486602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.487001 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:35.487149 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:35.487962 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 07:30:35.488258 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:35.489681 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.470184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:58543)
0926 07:30:35.470866 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.470903 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.472617 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.473410 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.473423 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.474030 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.474566 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.474581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.475690 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.475694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.476908 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.476919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.477760 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.477768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.477985 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.478660 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.478689 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.480828 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:35.485167 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.485176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.485191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.485800 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.486234 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.486239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.486243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.486465 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.487128 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:35.487136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:35.487539 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 07:30:35.487832 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 07:30:35.488356 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":254,"threads_started":1}
I20260926 07:30:35.491459 17365 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:35.491789 17365 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:35.492707 17365 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
I20260926 07:30:35.494391 17048 ts_manager.cc:194] Registered new tserver with Master: cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
I20260926 07:30:35.495342 17048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:58543
W20260926 07:30:35.813552 17368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:35.813896 17368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:35.814087 17368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:35.822257 17368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:35.822422 17368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:35.822607 17368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:35.832536 17368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:45773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=32985
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:35.833868 17368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:35.835368 17368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:35.848896 17376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:35.848979 17374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:35.848956 17373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:35.850108 17368 server_base.cc:1034] running on GCE node
I20260926 07:30:35.850801 17368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:35.851847 17368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:35.853048 17368 hybrid_clock.cc:648] HybridClock initialized: now 1790407835853015 us; error 35 us; skew 500 ppm
I20260926 07:30:35.853382 17368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:35 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:35.856489 17368 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:35.857919 17368 webserver.cc:460] Webserver started at http://127.2.147.195:32985/ using document root <none> and password file <none>
I20260926 07:30:35.858611 17368 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:35.858778 17368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:35.864257 17368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:35.867544 17383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.868960 17368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:35.869166 17368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "1ae74f082b7e440babd7163a4ec55357"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "ac3d6a3e9ea844daa95d5f2ad9a6c14a"
server_key_iv: "49b83f6faa1b42eeb9f9933701b94706"
server_key_version: "encryptionkey@0"
I20260926 07:30:35.870167 17368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:35.890662 17368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:35.895025 17368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:35.895385 17368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:35.896409 17368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:35.897715 17368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:35.897799 17368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.897881 17368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:35.897948 17368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:35.931363 17368 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:45773
I20260926 07:30:35.931416 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:45773 every 8 connection(s)
I20260926 07:30:35.932932 17368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:35.942790 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17368
Sep 26 07:30:35 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:35.955122 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.935628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:34441 (local address 127.2.147.254:41439)
0926 07:30:35.935819 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.935823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.937014 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 07:30:35.938929 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.938935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.939029 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.939168 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.940694 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.941438 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.943532 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.944380 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.944510 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.948347 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:35.948354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:35.948359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:35.948397 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:35.951882 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.952583 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.952589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.952594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.952756 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.953213 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.953217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.953219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.953507 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:35.953619 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:35.954546 (+ 927us) server_negotiation.cc:299] Negotiation successful
0926 07:30:35.954717 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:30:35.956472 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.936126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:34441)
0926 07:30:35.936851 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.936894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.938652 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.939434 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.939442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.939959 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.940512 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.940530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.941604 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.941609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.943321 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.943336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.944623 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.944652 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.944980 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.945781 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.945807 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.948138 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:35.952030 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.952036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.952048 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.952454 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.952869 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:35.952873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:35.952875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:35.953085 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:35.953640 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:35.953652 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:35.954111 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 07:30:35.954434 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 07:30:35.954989 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":265,"threads_started":1}
I20260926 07:30:35.957984 17496 heartbeater.cc:347] Connected to a master server at 127.2.147.254:41439
I20260926 07:30:35.958298 17496 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:35.959018 17496 heartbeater.cc:511] Master 127.2.147.254:41439 requested a full tablet report, sending...
I20260926 07:30:35.959334 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.943835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.2.147.254:41439)
0926 07:30:35.944486 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.944492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.944669 (+ 177us) server_negotiation.cc:407] Connection header received
0926 07:30:35.944845 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.944850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.944922 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.945044 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.946635 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.947558 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.949371 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.949826 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.949954 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.953154 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:35.953170 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:35.953176 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:35.953218 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:35.956569 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.957299 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.957307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.957314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.957431 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:35.957801 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:35.957807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:35.957810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:35.958092 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:35.958366 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:35.958669 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:30:35.958898 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":298,"threads_started":1}
I20260926 07:30:35.960829 17048 ts_manager.cc:194] Registered new tserver with Master: 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773)
I20260926 07:30:35.961911 17048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:34441
I20260926 07:30:35.962046 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:35.969059 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.963015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.2.147.254:41439)
0926 07:30:35.963195 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.963202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.963645 (+ 443us) server_negotiation.cc:407] Connection header received
0926 07:30:35.963871 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.963875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.963994 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.964140 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:35.964562 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.965330 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.966868 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.967359 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.967491 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.967981 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:35.968443 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:35.968617 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:35.968724 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 07:30:35.968804 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:30:35.969908 17048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:35.977273 17048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:36.001183 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.989081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.2.147.195:45773)
0926 07:30:35.989794 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.989804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.990001 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:30:35.990263 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.990270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.990480 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.990685 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:35.991465 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.993291 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.996984 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.998990 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.999123 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.999481 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.000403 (+ 922us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.000649 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":361,"threads_started":1}
I20260926 07:30:36.002341 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.988961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:45773 (local address 127.0.0.1:49822)
0926 07:30:35.989845 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.989898 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.990040 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.990708 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.990715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.990747 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:35.991190 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.991204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.993546 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.993555 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.996746 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.996767 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.999723 (+ 2956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.999732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.000041 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.000115 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 07:30:36.001555 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0926 07:30:36.001841 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":725,"threads_started":1}
I20260926 07:30:36.008411 17431 tablet_service.cc:1467] Processing CreateTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 (DEFAULT_TABLE table=test-table [id=7dcb6dc11cef45e890fbb0a4f037658e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:36.010037 17431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:36.011345 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.991554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57798)
0926 07:30:36.002645 (+ 11091us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.002667 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.002795 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.004592 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.004597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.004624 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:36.004803 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.004812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.006721 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.006727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.008745 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.008759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.010561 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.010568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.010761 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.010800 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:30:36.010869 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:30:36.011006 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10952,"thread_start_us":12441,"threads_started":1}
I20260926 07:30:36.012528 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.991993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.2.147.193:38201)
0926 07:30:36.003857 (+ 11864us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.003870 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.003908 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:30:36.004024 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.004033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.004360 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.004588 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:36.004970 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.006568 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.008927 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.010372 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.011725 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.011959 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.012029 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.012227 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11709,"thread_start_us":11608,"threads_started":1}
I20260926 07:30:36.013005 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.990511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37233 (local address 127.0.0.1:53906)
0926 07:30:35.991397 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.991416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.991512 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.005156 (+ 13644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.005161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.005186 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:36.005544 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.005557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.007480 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.007487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.009486 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.009500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.012205 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.012212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.012380 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.012412 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:30:36.012480 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:30:36.012638 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":1,"thread_start_us":574,"threads_started":1}
I20260926 07:30:36.013576 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.990831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.2.147.194:37233)
0926 07:30:36.004521 (+ 13690us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.004531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.004558 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:30:36.004647 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.004657 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.004947 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.005156 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:36.005759 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.007325 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.009789 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.011338 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.012806 (+ 1468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.013011 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.013078 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.013301 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13523,"thread_start_us":13419,"threads_started":1}
I20260926 07:30:36.016635 17168 tablet_service.cc:1467] Processing CreateTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 (DEFAULT_TABLE table=test-table [id=7dcb6dc11cef45e890fbb0a4f037658e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:36.017076 17300 tablet_service.cc:1467] Processing CreateTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 (DEFAULT_TABLE table=test-table [id=7dcb6dc11cef45e890fbb0a4f037658e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:36.018289 17300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:36.018301 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:36.026481 17512 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Bootstrap starting.
I20260926 07:30:36.030045 17512 tablet_bootstrap.cc:654] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:36.031242 17513 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Bootstrap starting.
I20260926 07:30:36.031343 17512 log.cc:824] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:36.034358 17513 tablet_bootstrap.cc:654] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:36.034626 17512 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: No bootstrap required, opened a new log
I20260926 07:30:36.034898 17512 ts_tablet_manager.cc:1397] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:30:36.035568 17513 log.cc:824] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:36.038268 17516 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Bootstrap starting.
I20260926 07:30:36.040014 17513 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: No bootstrap required, opened a new log
I20260926 07:30:36.040284 17513 ts_tablet_manager.cc:1397] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 07:30:36.041303 17516 tablet_bootstrap.cc:654] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:36.042378 17516 log.cc:824] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:36.045503 17516 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: No bootstrap required, opened a new log
I20260926 07:30:36.045765 17516 ts_tablet_manager.cc:1397] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 07:30:36.045646 17512 raft_consensus.cc:348] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.046089 17512 raft_consensus.cc:374] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:36.046198 17512 raft_consensus.cc:729] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae74f082b7e440babd7163a4ec55357, State: Initialized, Role: FOLLOWER
I20260926 07:30:36.046803 17512 consensus_queue.cc:260] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.047955 17496 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:36.048542 17512 ts_tablet_manager.cc:1428] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 07:30:36.050683 17513 raft_consensus.cc:348] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.051141 17513 raft_consensus.cc:374] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:36.051265 17513 raft_consensus.cc:729] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5aa0ded97c4b6681bcdd4374d65ae0, State: Initialized, Role: FOLLOWER
I20260926 07:30:36.051887 17513 consensus_queue.cc:260] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.053264 17365 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:36.053833 17513 ts_tablet_manager.cc:1428] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 07:30:36.055670 17516 raft_consensus.cc:348] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.056136 17516 raft_consensus.cc:374] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:36.056241 17516 raft_consensus.cc:729] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f43d1dbace4808a6a07c46a8430467, State: Initialized, Role: FOLLOWER
I20260926 07:30:36.056757 17233 heartbeater.cc:503] Master 127.2.147.254:41439 was elected leader, sending a full tablet report...
I20260926 07:30:36.056861 17516 consensus_queue.cc:260] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.058614 17516 ts_tablet_manager.cc:1428] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 07:30:36.111541 17518 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:36.111850 17518 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.113579 17518 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233), 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201)
I20260926 07:30:36.120373 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.114036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:55933 (local address 127.2.147.194:37233)
0926 07:30:36.114230 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.114234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.114253 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:30:36.114456 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.114461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.114546 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.114658 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:36.115257 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.116368 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.118372 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.119581 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.119706 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.119756 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.120087 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.120179 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:30:36.120479 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.113946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37233 (local address 127.2.147.195:55933)
0926 07:30:36.114150 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.114180 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.114304 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.114709 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.114717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.114758 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:36.115073 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.115086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.116511 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.116518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.118209 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.118221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.119732 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.119738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.119927 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.119960 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:30:36.120060 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:30:36.120195 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":23}
I20260926 07:30:36.121447 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.114668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:59677 (local address 127.2.147.193:38201)
0926 07:30:36.114891 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.114899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.115431 (+ 532us) server_negotiation.cc:407] Connection header received
0926 07:30:36.115591 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.115596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.115670 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.115741 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:36.116283 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.117324 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.119455 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.120597 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.120765 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.120847 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.121173 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.121267 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:30:36.121623 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.114400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.2.147.195:59677)
0926 07:30:36.115313 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.115334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.115460 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.115787 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.115791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.115816 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:36.116138 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.116148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.117477 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.117481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.119302 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.119316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.120819 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.120825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.120995 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.121018 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:30:36.121129 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:30:36.121265 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":19,"thread_start_us":523,"threads_started":1}
I20260926 07:30:36.121619 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4950a9a5d0d48acbf5bdebe89dd8aa4" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" is_pre_election: true
I20260926 07:30:36.122220 17320 raft_consensus.cc:2454] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 0.
I20260926 07:30:36.122777 17188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4950a9a5d0d48acbf5bdebe89dd8aa4" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f43d1dbace4808a6a07c46a8430467" is_pre_election: true
I20260926 07:30:36.122895 17384 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0; no voters:
I20260926 07:30:36.123309 17518 raft_consensus.cc:2781] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:36.123291 17188 raft_consensus.cc:2454] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 0.
I20260926 07:30:36.123431 17518 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:36.123550 17518 raft_consensus.cc:3037] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:36.125870 17518 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.126526 17518 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 election: Requested vote from peers cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233), 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201)
I20260926 07:30:36.126912 17320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4950a9a5d0d48acbf5bdebe89dd8aa4" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
I20260926 07:30:36.127002 17188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4950a9a5d0d48acbf5bdebe89dd8aa4" candidate_uuid: "1ae74f082b7e440babd7163a4ec55357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f43d1dbace4808a6a07c46a8430467"
I20260926 07:30:36.127115 17320 raft_consensus.cc:3037] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:36.127264 17188 raft_consensus.cc:3037] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:36.129429 17188 raft_consensus.cc:2454] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 1.
I20260926 07:30:36.129438 17320 raft_consensus.cc:2454] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae74f082b7e440babd7163a4ec55357 in term 1.
I20260926 07:30:36.129799 17384 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467, 1ae74f082b7e440babd7163a4ec55357; no voters:
I20260926 07:30:36.130125 17518 raft_consensus.cc:2781] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:36.130566 17518 raft_consensus.cc:686] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae74f082b7e440babd7163a4ec55357, State: Running, Role: LEADER
I20260926 07:30:36.131085 17518 consensus_queue.cc:237] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:36.135839 17047 catalog_manager.cc:5495] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "1ae74f082b7e440babd7163a4ec55357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:36.153882 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.148116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.2.147.195:45773)
0926 07:30:36.148324 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.148330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.148432 (+ 102us) server_negotiation.cc:407] Connection header received
0926 07:30:36.148694 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.148698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.148777 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.148931 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:36.149294 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.150098 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.151732 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.152165 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.152301 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.152790 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:36.153302 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:36.153469 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.153527 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.153609 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:30:36.162315 17320 raft_consensus.cc:1261] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Refusing update from remote peer 1ae74f082b7e440babd7163a4ec55357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:36.162315 17188 raft_consensus.cc:1261] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Refusing update from remote peer 1ae74f082b7e440babd7163a4ec55357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:36.163362 17522 consensus_queue.cc:1035] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:36.163738 17518 consensus_queue.cc:1035] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:36.173750 17524 mvcc.cc:205] Tried to move back new op lower bound from 7333510496910061568 to 7333510496797433856. Current Snapshot: MvccSnapshot[applied={T|T < 7333510496910061568}]
I20260926 07:30:36.174893 17526 mvcc.cc:205] Tried to move back new op lower bound from 7333510496910061568 to 7333510496797433856. Current Snapshot: MvccSnapshot[applied={T|T < 7333510496910061568}]
I20260926 07:30:36.191932 17047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:36.192273 17047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:36.195930 17047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Sending DeleteTablet for 3 replicas of tablet d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:36.197073 17431 tablet_service.cc:1514] Processing DeleteTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:36 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:49822
I20260926 07:30:36.197291 17300 tablet_service.cc:1514] Processing DeleteTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:36 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53906
I20260926 07:30:36.197307 17168 tablet_service.cc:1514] Processing DeleteTablet for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:36 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:57798
I20260926 07:30:36.198575 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17085
I20260926 07:30:36.212904 17536 tablet_replica.cc:330] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: stopping tablet replica
I20260926 07:30:36.213457 17536 raft_consensus.cc:2229] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:36.214015 17536 raft_consensus.cc:2258] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:36.216450 17536 ts_tablet_manager.cc:1905] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:36.223608 17536 ts_tablet_manager.cc:1918] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:36.223790 17536 log.cc:1196] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:36.224407 17536 ts_tablet_manager.cc:1939] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P cc5aa0ded97c4b6681bcdd4374d65ae0: Deleting consensus metadata
I20260926 07:30:36.226346 17032 catalog_manager.cc:4841] TS cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 (table test-table [id=7dcb6dc11cef45e890fbb0a4f037658e]) successfully deleted
I20260926 07:30:36.231307 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.199516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.2.147.193:38201)
0926 07:30:36.199695 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.199701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.200134 (+ 433us) server_negotiation.cc:407] Connection header received
0926 07:30:36.200216 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.200225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.200332 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.200444 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.203084 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.204704 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.207985 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.208946 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.209100 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.220682 (+ 11582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:36.220712 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:36.220737 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:36.220801 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:36.226398 (+ 5597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.227822 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.227833 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.227836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.228012 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.228552 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.228556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.228560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.229083 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:36.229415 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.230626 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.230873 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:30:36.231983 17228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:36.232969 17534 tablet_replica.cc:330] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: stopping tablet replica
I20260926 07:30:36.233745 17534 raft_consensus.cc:2229] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:36.234679 17534 raft_consensus.cc:2258] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:36.238497 17534 ts_tablet_manager.cc:1905] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:36.247712 17534 ts_tablet_manager.cc:1918] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:36.247922 17534 log.cc:1196] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:36.248556 17534 ts_tablet_manager.cc:1939] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 1ae74f082b7e440babd7163a4ec55357: Deleting consensus metadata
I20260926 07:30:36.250721 17032 catalog_manager.cc:4841] TS 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): tablet d4950a9a5d0d48acbf5bdebe89dd8aa4 (table test-table [id=7dcb6dc11cef45e890fbb0a4f037658e]) successfully deleted
I20260926 07:30:36.330276 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17085
I20260926 07:30:36.348240 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17237
W20260926 07:30:36.348297 17032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.147.193:38201 (error 108)
I20260926 07:30:36.350183 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.349672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57818)
0926 07:30:36.349819 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.349962 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:30:36.350564 17032 catalog_manager.cc:4584] TS 02f43d1dbace4808a6a07c46a8430467 (127.2.147.193:38201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4950a9a5d0d48acbf5bdebe89dd8aa4: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
I20260926 07:30:36.369728 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.349172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.2.147.194:37233)
0926 07:30:36.349369 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.349377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.349660 (+ 283us) server_negotiation.cc:407] Connection header received
0926 07:30:36.349889 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.349896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.350002 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.350077 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.352075 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.352795 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.354767 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.355144 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.355303 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.363675 (+ 8372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:36.363692 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:36.363711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:36.363752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:36.366872 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.367749 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.367756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.367758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.367881 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.368307 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.368310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.368312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.368678 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:36.368905 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.369252 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.369419 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:30:36.370458 17360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:36.404309 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.403734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57820)
0926 07:30:36.403984 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.404102 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 07:30:36.405486 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.405052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57826)
0926 07:30:36.405226 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.405313 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:30:36.460018 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17237
I20260926 07:30:36.466481 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.465868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57836)
0926 07:30:36.466110 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.466242 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:30:36.467736 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.467317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57842)
0926 07:30:36.467493 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.467585 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:30:36.478408 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17368
I20260926 07:30:36.493170 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.479178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.2.147.195:45773)
0926 07:30:36.479393 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.479399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.479529 (+ 130us) server_negotiation.cc:407] Connection header received
0926 07:30:36.479762 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.479766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.479827 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.479936 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.481229 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.482042 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.483440 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.484064 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.484234 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.486680 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:36.486697 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:36.486713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:36.486754 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:36.489938 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.490653 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.490662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.490668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.490826 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:36.491499 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:36.491503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:36.491505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:36.491873 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:36.492272 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:36.492502 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 07:30:36.492729 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:30:36.493928 17491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:36.548995 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.548518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57848)
0926 07:30:36.548724 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.548823 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:30:36.550137 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.549760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57854)
0926 07:30:36.549946 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.550023 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:30:36.616333 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17368
I20260926 07:30:36.636003 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17015
I20260926 07:30:36.637120 17078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:36.750623 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17015
I20260926 07:30:36.750793 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.750050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:38201 (local address 127.0.0.1:57868)
0926 07:30:36.750395 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.750576 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:38201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 07:30:36.772461 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--webserver_interface=127.2.147.254
--webserver_port=38065
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:37.099141 17542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:37.099560 17542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:37.099725 17542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:37.110370 17542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:37.110532 17542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:37.110620 17542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:37.110702 17542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:37.110764 17542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:37.121037 17542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=38065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:37.122402 17542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:37.123805 17542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:37.135366 17547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:37.135339 17548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:37.135311 17550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:37.136509 17542 server_base.cc:1034] running on GCE node
I20260926 07:30:37.137440 17542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:37.138924 17542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:37.140177 17542 hybrid_clock.cc:648] HybridClock initialized: now 1790407837140140 us; error 41 us; skew 500 ppm
I20260926 07:30:37.140560 17542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:37 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:37.143774 17542 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:37.145344 17542 webserver.cc:460] Webserver started at http://127.2.147.254:38065/ using document root <none> and password file <none>
I20260926 07:30:37.146035 17542 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:37.146193 17542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:37.151638 17542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:37.154580 17557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:37.156035 17542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:37.156185 17542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f71a8bce1c2749de9f5a55e988760640"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "a50506db129bdb0915ae90664b4b79bb"
server_key_iv: "fd6348a40d2e4228e6fe19bfb108d573"
server_key_version: "encryptionkey@0"
I20260926 07:30:37.157105 17542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:37.170603 17542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:37.175019 17542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:37.175318 17542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:37.193886 17542 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41439
I20260926 07:30:37.193909 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41439 every 8 connection(s)
I20260926 07:30:37.195415 17542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:37.202119 17609 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:37.205180 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17542
I20260926 07:30:37.205437 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap starting.
I20260926 07:30:37.206044 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:38201
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=34843
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:37.226207 17609 log.cc:824] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:37.242340 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:37.242906 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640: Bootstrap complete.
I20260926 07:30:37.251973 17609 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:37.252897 17609 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Initialized, Role: FOLLOWER
I20260926 07:30:37.253484 17609 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:37.253737 17609 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:37.253870 17609 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:37.254052 17609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:30:37.257856 17609 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:37.258397 17609 leader_election.cc:302] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71a8bce1c2749de9f5a55e988760640; no voters:
I20260926 07:30:37.259097 17609 leader_election.cc:288] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:30:37.259245 17613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:30:37.260433 17613 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [term 3 LEADER]: Becoming Leader. State: Replica: f71a8bce1c2749de9f5a55e988760640, State: Running, Role: LEADER
I20260926 07:30:37.261075 17613 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } }
I20260926 07:30:37.261498 17609 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:37.271134 17614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:37.271538 17614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:37.277297 17615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71a8bce1c2749de9f5a55e988760640. Latest consensus state: current_term: 3 leader_uuid: "f71a8bce1c2749de9f5a55e988760640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71a8bce1c2749de9f5a55e988760640" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41439 } } }
I20260926 07:30:37.277681 17615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71a8bce1c2749de9f5a55e988760640 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:37.279222 17623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:37.287976 17623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:37.289501 17623 catalog_manager.cc:1259] Loaded cluster ID: a4070613dbdf441dbb59a3abadd2b217
I20260926 07:30:37.289669 17623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:30:37.291800 17623 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f16ec88c9ad google::LogMessage::Fail() at ??:0
@ 0x7f16ec88f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f16ec88c4de google::LogMessage::Flush() at ??:0
@ 0x7f16ec88da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f17022bf512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f17022bc976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1702344610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1702344439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f16ef4d8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f16ef4db91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f16ef4db799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f16ef4b6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f16f27906db start_thread at ??:0
@ 0x7f16eb72661f clone at ??:0
W20260926 07:30:37.574618 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:37.574963 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:37.575143 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:37.582610 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:37.582760 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:37.582948 17611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:37.593156 17611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:38201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=34843
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:37.594513 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:37.595990 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:37.609472 17651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:37.609539 17654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:37.609499 17652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:37.610357 17611 server_base.cc:1034] running on GCE node
I20260926 07:30:37.611020 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:37.612104 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:37.613335 17611 hybrid_clock.cc:648] HybridClock initialized: now 1790407837613279 us; error 46 us; skew 500 ppm
I20260926 07:30:37.613655 17611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:37 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:37.616844 17611 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:37.618386 17611 webserver.cc:460] Webserver started at http://127.2.147.193:34843/ using document root <none> and password file <none>
I20260926 07:30:37.619069 17611 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:37.619262 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:37.624375 17611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:30:37.627189 17661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:37.628573 17611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:37.628744 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "02f43d1dbace4808a6a07c46a8430467"
format_stamp: "Formatted at 2026-09-26 07:30:31 on dist-test-slave-1dwd"
server_key: "18f1c4b4744ce3c9c34cb79669bdee55"
server_key_iv: "a19d64c1436a95634f453288efa8f646"
server_key_version: "encryptionkey@0"
I20260926 07:30:37.629688 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:37.654340 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:37.658973 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:37.659288 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:37.660302 17611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:37.662094 17668 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 07:30:37.667991 17611 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:30:37.668140 17611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 07:30:37.668285 17611 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:30:37.671424 17611 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:30:37.671536 17611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:37.671873 17668 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Bootstrap starting.
I20260926 07:30:37.699469 17668 log.cc:824] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:37.712427 17668 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:37.713096 17668 tablet_bootstrap.cc:492] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Bootstrap complete.
I20260926 07:30:37.714687 17668 ts_tablet_manager.cc:1397] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Time spent bootstrapping tablet: real 0.043s user 0.035s sys 0.004s
I20260926 07:30:37.716142 17611 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:38201
I20260926 07:30:37.716166 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:38201 every 8 connection(s)
I20260926 07:30:37.717518 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:37.720464 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17611
I20260926 07:30:37.721326 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:37233
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=35283
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:37.723533 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.720681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:37795)
0926 07:30:37.721944 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.722272 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":483,"threads_started":1}
W20260926 07:30:37.724272 17776 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:41439 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:41439: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
I20260926 07:30:37.723344 17668 raft_consensus.cc:348] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:37.724678 17668 raft_consensus.cc:729] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f43d1dbace4808a6a07c46a8430467, State: Initialized, Role: FOLLOWER
I20260926 07:30:37.725268 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.724847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:58301)
0926 07:30:37.725030 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.725139 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:30:37.725530 17668 consensus_queue.cc:260] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:37.726372 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.725903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:42113)
0926 07:30:37.726097 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.726200 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:30:37.726792 17776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:30:37.728469 17668 ts_tablet_manager.cc:1428] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260926 07:30:37.728504 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.728118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:49341)
0926 07:30:37.728281 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.728364 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:30:37.729548 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.729235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:35775)
0926 07:30:37.729373 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.729445 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:30:38.046326 17780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:38.046612 17780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:38.046734 17780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:38.054070 17780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:38.054207 17780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:38.054399 17780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:38.064419 17780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:37233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=35283
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:38.065820 17780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:38.067238 17780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:38.080863 17785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:38.080907 17786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:38.080928 17788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:38.081712 17780 server_base.cc:1034] running on GCE node
I20260926 07:30:38.082366 17780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:38.083397 17780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:38.084604 17780 hybrid_clock.cc:648] HybridClock initialized: now 1790407838084561 us; error 31 us; skew 500 ppm
I20260926 07:30:38.084947 17780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:38 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:38.088156 17780 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:38.089591 17780 webserver.cc:460] Webserver started at http://127.2.147.194:35283/ using document root <none> and password file <none>
I20260926 07:30:38.090312 17780 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:38.090446 17780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:38.096009 17780 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:38.098706 17795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.100117 17780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:38.100299 17780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "04983ce7c07122d9461d7f7a7260cec3"
server_key_iv: "483380d0778e45807204f4f61d448778"
server_key_version: "encryptionkey@0"
I20260926 07:30:38.101332 17780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:38.132839 17780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:38.137120 17780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:38.137415 17780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:38.138449 17780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:38.139818 17780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:38.139940 17780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.140077 17780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:38.140172 17780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.172726 17780 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37233
I20260926 07:30:38.172785 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37233 every 8 connection(s)
I20260926 07:30:38.174221 17780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:38.176359 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17780
I20260926 07:30:38.177160 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:45773
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:41439
--webserver_port=32985
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:38.179533 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.177576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:59027)
0926 07:30:38.178370 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.178601 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
W20260926 07:30:38.180312 17908 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:41439 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:41439: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
I20260926 07:30:38.181154 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.180827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:57537)
0926 07:30:38.180979 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.181054 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 07:30:38.181983 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.181662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:41027)
0926 07:30:38.181801 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.181882 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:30:38.182192 17908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:30:38.500839 17911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:38.501199 17911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:38.501360 17911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:38.509356 17911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:38.509516 17911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:38.509690 17911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:38.519461 17911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:45773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=32985
--tserver_master_addrs=127.2.147.254:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:38.520771 17911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:38.522109 17911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:38.535106 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:38.535050 17917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:38.535154 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:38.535953 17911 server_base.cc:1034] running on GCE node
I20260926 07:30:38.536582 17911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:38.537514 17911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:38.538741 17911 hybrid_clock.cc:648] HybridClock initialized: now 1790407838538680 us; error 54 us; skew 500 ppm
I20260926 07:30:38.539065 17911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:38 dist-test-slave-1dwd krb5kdc[16465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:38.542193 17911 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:38.543664 17911 webserver.cc:460] Webserver started at http://127.2.147.195:32985/ using document root <none> and password file <none>
I20260926 07:30:38.544329 17911 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:38.544425 17911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:38.549562 17911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:30:38.552129 17926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.553435 17911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:38.553610 17911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "1ae74f082b7e440babd7163a4ec55357"
format_stamp: "Formatted at 2026-09-26 07:30:32 on dist-test-slave-1dwd"
server_key: "ac3d6a3e9ea844daa95d5f2ad9a6c14a"
server_key_iv: "49b83f6faa1b42eeb9f9933701b94706"
server_key_version: "encryptionkey@0"
I20260926 07:30:38.554559 17911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:38.572726 17911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:38.576706 17911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:38.576992 17911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:38.577971 17911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:38.579340 17911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:38.579458 17911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.579596 17911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:38.579674 17911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:38.613016 17911 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:45773
I20260926 07:30:38.613106 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:45773 every 8 connection(s)
I20260926 07:30:38.614528 17911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:38.619408 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.617531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:44949)
0926 07:30:38.618234 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.618460 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260926 07:30:38.619694 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 17911
W20260926 07:30:38.620225 18039 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:41439 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:41439: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
I20260926 07:30:38.621196 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.620756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:54481)
0926 07:30:38.620952 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.621054 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:30:38.621999 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.621709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:33533)
0926 07:30:38.621825 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.621892 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:30:38.622227 18039 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:30:38.731148 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.730381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:48849)
0926 07:30:38.730852 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.730946 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260926 07:30:39.183570 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.182748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:47097)
0926 07:30:39.183189 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.183334 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260926 07:30:39.422226 18045 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:39.422458 18045 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:39.423981 18045 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
Sep 26 07:30:39 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
Sep 26 07:30:39 dist-test-slave-1dwd krb5kdc[16465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:30:39.448338 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.424460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:45061 (local address 127.2.147.195:45773)
0926 07:30:39.426724 (+ 2264us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:39.426744 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:39.426789 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:30:39.429060 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:39.429072 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:39.429595 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:39.429975 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:39.432235 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.433484 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.435156 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.435832 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.436025 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.439831 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:39.439839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:39.439853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:39.439891 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:39.443709 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:39.444827 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:39.444836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:39.444844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:39.445028 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:39.445607 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:39.445612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:39.445615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:39.446270 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:39.446443 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:39.447498 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0926 07:30:39.447912 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":1965,"threads_started":1}
I20260926 07:30:39.448338 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.425571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:48687 (local address 127.2.147.194:37233)
0926 07:30:39.427024 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:39.427038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:39.427079 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:30:39.429255 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:39.429265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:39.429595 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:39.429975 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:39.432238 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.433571 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.435157 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.435828 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.436028 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.440097 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:39.440103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:39.440115 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:39.440156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:39.443709 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:39.444827 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:39.444836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:39.444844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:39.445028 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:39.445607 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:39.445612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:39.445615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:39.446273 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:39.446446 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:39.447498 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 07:30:39.447912 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1081,"threads_started":1}
I20260926 07:30:39.448342 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.424298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:45773 (local address 127.2.147.193:45061)
0926 07:30:39.424875 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.424913 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:39.425549 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:39.430402 (+ 4853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:39.430407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:39.431271 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:39.432032 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.432053 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.433661 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.433666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.435006 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.435019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.436073 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.436080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.436352 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.437200 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:39.437216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:39.439545 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:39.443874 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:39.443884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:39.443899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:39.444690 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:39.445161 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:39.445166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:39.445170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:39.445484 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:39.446422 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:39.446429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:39.446912 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 07:30:39.447251 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 07:30:39.447819 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":200,"spinlock_wait_cycles":17920,"thread_start_us":303,"threads_started":1}
I20260926 07:30:39.448366 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.425041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37233 (local address 127.2.147.193:48687)
0926 07:30:39.426350 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.426367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:39.426483 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:39.430141 (+ 3658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:39.430153 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:39.431272 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:39.432032 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.432052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.433736 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.433740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.435008 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.435019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.436043 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.436053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.436350 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.437128 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:39.437152 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:39.439841 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:39.443874 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:39.443884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:39.443899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:39.444707 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:39.445161 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:39.445166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:39.445170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:39.445484 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:39.446422 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:39.446430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:39.446912 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 07:30:39.447251 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 07:30:39.447982 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":354,"spinlock_wait_cycles":48384,"thread_start_us":1029,"threads_started":1}
W20260926 07:30:39.451771 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
W20260926 07:30:39.452193 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:39.452342 17662 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467; no voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0
I20260926 07:30:39.452757 18045 raft_consensus.cc:2726] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:30:39.623652 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.622798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:39429)
0926 07:30:39.623301 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.623438 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":214,"threads_started":1}
I20260926 07:30:39.732429 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.732028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:39027)
0926 07:30:39.732185 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.732285 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:30:40.185086 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.184423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:32813)
0926 07:30:40.184852 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.184936 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":227,"threads_started":1}
I20260926 07:30:40.625058 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.624464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:45453)
0926 07:30:40.624846 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.624920 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260926 07:30:40.734169 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.733367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:40587)
0926 07:30:40.733849 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.733937 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":220,"threads_started":1}
I20260926 07:30:41.186759 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.186067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:48419)
0926 07:30:41.186517 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.186597 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
I20260926 07:30:41.292296 18055 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:41.292518 18055 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:41.293241 18055 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
W20260926 07:30:41.294447 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
W20260926 07:30:41.294690 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:41.294802 17662 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467; no voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0
I20260926 07:30:41.295106 18055 raft_consensus.cc:2726] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:30:41.626531 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.625894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:49385)
0926 07:30:41.626304 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.626380 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
I20260926 07:30:41.735873 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.735042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:39879)
0926 07:30:41.735555 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.735684 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260926 07:30:42.188458 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.187758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:56615)
0926 07:30:42.188192 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.188282 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260926 07:30:42.628273 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.627413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:37159)
0926 07:30:42.627986 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.628066 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":339,"threads_started":1}
I20260926 07:30:42.737766 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.736924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:46137)
0926 07:30:42.737491 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.737584 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
I20260926 07:30:43.190184 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.189373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:50449)
0926 07:30:43.189941 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.190026 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":331,"threads_started":1}
I20260926 07:30:43.629932 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.629135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:41761)
0926 07:30:43.629670 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.629771 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
I20260926 07:30:43.660106 18063 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:43.660290 18063 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:43.660840 18063 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
W20260926 07:30:43.661725 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
W20260926 07:30:43.661943 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:43.662070 17662 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467; no voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0
I20260926 07:30:43.662338 18063 raft_consensus.cc:2726] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:30:43.739516 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.738645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:39725)
0926 07:30:43.739065 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.739165 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260926 07:30:44.191747 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.191008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:60397)
0926 07:30:44.191485 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.191564 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":279,"threads_started":1}
I20260926 07:30:44.631834 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.630915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:52799)
0926 07:30:44.631479 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.631581 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260926 07:30:44.741219 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.740468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:45417)
0926 07:30:44.740903 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.741020 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260926 07:30:45.193434 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.192663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:49787)
0926 07:30:45.193153 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.193246 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":234,"threads_started":1}
I20260926 07:30:45.633862 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.632847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:48677)
0926 07:30:45.633485 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.633608 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":312,"threads_started":1}
I20260926 07:30:45.742853 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.742161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:58989)
0926 07:30:45.742583 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.742678 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260926 07:30:46.194960 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.194311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:38269)
0926 07:30:46.194711 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:46.194792 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260926 07:30:46.635576 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.634799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:37199)
0926 07:30:46.635290 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:46.635373 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260926 07:30:46.744400 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.743729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:58385)
0926 07:30:46.744157 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:46.744236 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260926 07:30:47.196518 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.195806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:37253)
0926 07:30:47.196280 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.196366 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260926 07:30:47.637082 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.636402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:46585)
0926 07:30:47.636859 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.636937 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260926 07:30:47.746006 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.745344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:51771)
0926 07:30:47.745753 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.745832 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260926 07:30:48.198287 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:48.197576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:46301)
0926 07:30:48.198029 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:48.198120 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":269,"threads_started":1}
I20260926 07:30:48.540827 18078 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:48.541049 18078 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:48.541646 18078 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
W20260926 07:30:48.542614 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
W20260926 07:30:48.542871 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:48.542980 17662 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467; no voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0
I20260926 07:30:48.543298 18078 raft_consensus.cc:2726] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:30:48.638612 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:48.637953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:60385)
0926 07:30:48.638367 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:48.638444 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":231,"threads_started":1}
I20260926 07:30:48.747684 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:48.746850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:47965)
0926 07:30:48.747360 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:48.747490 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":261,"threads_started":1}
I20260926 07:30:49.199997 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.199298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:54579)
0926 07:30:49.199747 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:49.199836 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":214,"threads_started":1}
I20260926 07:30:49.640206 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.639482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:52465)
0926 07:30:49.639960 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:49.640042 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":214,"threads_started":1}
I20260926 07:30:49.749269 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.748540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:55493)
0926 07:30:49.748979 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:49.749060 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":252,"threads_started":1}
I20260926 07:30:50.201959 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.201117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:33555)
0926 07:30:50.201578 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.201740 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260926 07:30:50.641847 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.641050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:43419)
0926 07:30:50.641584 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.641683 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
I20260926 07:30:50.750963 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.750266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:44793)
0926 07:30:50.750699 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.750795 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260926 07:30:51.203868 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:51.203032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:42389)
0926 07:30:51.203591 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:51.203706 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":313,"threads_started":1}
I20260926 07:30:51.643500 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:51.642721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:56691)
0926 07:30:51.643190 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:51.643342 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260926 07:30:51.752542 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:51.751840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:56117)
0926 07:30:51.752279 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:51.752356 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":228,"threads_started":1}
I20260926 07:30:52.205651 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:52.204866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:44759)
0926 07:30:52.205369 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:52.205461 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":269,"threads_started":1}
I20260926 07:30:52.645135 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:52.644393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:40543)
0926 07:30:52.644851 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:52.644945 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260926 07:30:52.754410 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:52.753707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.193:34283)
0926 07:30:52.754158 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:52.754239 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260926 07:30:53.207609 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:53.206820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.194:56763)
0926 07:30:53.207364 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:53.207443 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":207,"threads_started":1}
I20260926 07:30:53.289886 18094 raft_consensus.cc:482] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:53.290068 18094 raft_consensus.cc:504] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ae74f082b7e440babd7163a4ec55357" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 45773 } } peers { permanent_uuid: "cc5aa0ded97c4b6681bcdd4374d65ae0" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37233 } } peers { permanent_uuid: "02f43d1dbace4808a6a07c46a8430467" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 38201 } }
I20260926 07:30:53.290629 18094 leader_election.cc:288] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773), cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233)
W20260926 07:30:53.291728 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ae74f082b7e440babd7163a4ec55357 (127.2.147.195:45773): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
W20260926 07:30:53.291957 17662 leader_election.cc:341] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc5aa0ded97c4b6681bcdd4374d65ae0 (127.2.147.194:37233): Not found: Tablet not found: d4950a9a5d0d48acbf5bdebe89dd8aa4
I20260926 07:30:53.292084 17662 leader_election.cc:302] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f43d1dbace4808a6a07c46a8430467; no voters: 1ae74f082b7e440babd7163a4ec55357, cc5aa0ded97c4b6681bcdd4374d65ae0
I20260926 07:30:53.292366 18094 raft_consensus.cc:2726] T d4950a9a5d0d48acbf5bdebe89dd8aa4 P 02f43d1dbace4808a6a07c46a8430467 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:30:53.629467 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17611
I20260926 07:30:53.647014 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:53.646268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:41439 (local address 127.2.147.195:46479)
0926 07:30:53.646750 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:53.646870 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":264,"threads_started":1}
I20260926 07:30:53.650261 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:53.630582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.2.147.193:38201)
0926 07:30:53.631193 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:53.631200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:53.631273 (+ 73us) server_negotiation.cc:407] Connection header received
0926 07:30:53.631344 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:53.631350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:53.631572 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:53.631714 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:53.633059 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.634141 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.635410 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.636155 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.636255 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:53.644733 (+ 8478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:53.644746 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:53.644760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:53.644795 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:53.647944 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.648572 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.648578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.648581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.648720 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.649045 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.649047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.649049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.649373 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:53.649633 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:53.649822 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:30:53.650010 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":294,"threads_started":1}
I20260926 07:30:53.651513 17771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:53.741848 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17611
I20260926 07:30:53.756704 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17780
I20260926 07:30:53.769668 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:53.757434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.2.147.194:37233)
0926 07:30:53.758042 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:53.758046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:53.758064 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:30:53.758125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:53.758129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:53.758189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:53.758286 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:53.759708 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.760467 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.761628 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.762061 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.762179 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:53.764734 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:53.764739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:53.764743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:53.764774 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:53.767507 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.768083 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.768087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.768089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.768226 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.768548 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.768551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.768553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.768822 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:53.769008 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:53.769210 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 07:30:53.769370 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":410,"threads_started":1}
I20260926 07:30:53.770480 17903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:53.853169 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17780
I20260926 07:30:53.870642 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 17911
I20260926 07:30:53.884400 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:53.871624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.2.147.195:45773)
0926 07:30:53.872289 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:53.872293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:53.872311 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:30:53.872369 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:53.872373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:53.872443 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:53.872540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:53.873902 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.874723 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.876162 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:53.876620 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:53.876727 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:53.879484 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:53.879489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:53.879493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:53.879529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:53.882005 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.882676 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.882679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.882682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.882814 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:53.883153 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:53.883156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:53.883158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:53.883479 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:53.883683 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:53.883928 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:30:53.884093 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":389,"threads_started":1}
I20260926 07:30:53.885159 18034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:53.970120 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 17911
2026-09-26T07:30:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24502 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:54 dist-test-slave-1dwd krb5kdc[18104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:54 dist-test-slave-1dwd krb5kdc[18104](info): set up 2 sockets
Sep 26 07:30:54 dist-test-slave-1dwd krb5kdc[18104](info): commencing operation
krb5kdc: starting...
W20260926 07:30:55.091816 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:30:55 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:30:55.334030 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:55.657016 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:55.657321 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:55.657506 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:55.664800 18117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:30:55.664909 18117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:30:55.665007 18117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:30:55.665117 18117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:55.665210 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:55.665334 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:55.665413 18117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:55.665499 18117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:55.675477 18117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:55.676824 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:55.678174 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:55.690569 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:55.690630 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:55.690651 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:55.691561 18117 server_base.cc:1034] running on GCE node
I20260926 07:30:55.692358 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:55.693753 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:55.695021 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790407855694954 us; error 38 us; skew 500 ppm
I20260926 07:30:55.695458 18117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:55 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:55.698621 18117 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:30:55.700090 18117 webserver.cc:460] Webserver started at http://127.2.147.254:37917/ using document root <none> and password file <none>
I20260926 07:30:55.700791 18117 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:55.700927 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:55.701244 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:55.704276 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "1912f4d4369e46e2be3083c9ee41a2ed"
format_stamp: "Formatted at 2026-09-26 07:30:55 on dist-test-slave-1dwd"
server_key: "c134087d1ca7e793b20a5bfea5c87786"
server_key_iv: "440ecc678df25f074ba13855bc283e03"
server_key_version: "encryptionkey@0"
I20260926 07:30:55.705200 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "1912f4d4369e46e2be3083c9ee41a2ed"
format_stamp: "Formatted at 2026-09-26 07:30:55 on dist-test-slave-1dwd"
server_key: "c134087d1ca7e793b20a5bfea5c87786"
server_key_iv: "440ecc678df25f074ba13855bc283e03"
server_key_version: "encryptionkey@0"
I20260926 07:30:55.710503 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:30:55.714016 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:55.715387 18117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:55.715572 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "1912f4d4369e46e2be3083c9ee41a2ed"
format_stamp: "Formatted at 2026-09-26 07:30:55 on dist-test-slave-1dwd"
server_key: "c134087d1ca7e793b20a5bfea5c87786"
server_key_iv: "440ecc678df25f074ba13855bc283e03"
server_key_version: "encryptionkey@0"
I20260926 07:30:55.715818 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:55.735106 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:55.739053 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:55.739369 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:55.757795 18117 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34279
I20260926 07:30:55.757824 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34279 every 8 connection(s)
I20260926 07:30:55.759159 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:55.762909 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:55.766525 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18117
I20260926 07:30:55.766757 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:30:55.767570 2639 external_mini_cluster.cc:1428] Setting key eb1e2257368dcdb9982071d48fe25dac
I20260926 07:30:55.776522 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap starting.
I20260926 07:30:55.780439 18184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:55.781809 18184 log.cc:824] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:30:55 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:55.785027 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: No bootstrap required, opened a new log
I20260926 07:30:55.791612 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:55.769750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.2.147.254:34279)
0926 07:30:55.770542 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:55.770554 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:55.770596 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:30:55.772327 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:55.772350 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:55.772778 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:55.773116 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:55.774913 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:55.776284 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:55.778118 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:55.778789 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:55.778918 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:55.784155 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:55.784163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:55.784178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:55.784218 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:55.787482 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:55.788317 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:55.788326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:55.788332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:55.788494 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:55.788897 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:55.788900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:55.788902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:55.789406 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:55.789637 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:55.790082 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:30:55.790474 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":377,"threads_started":1}
I20260926 07:30:55.794206 18184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:30:55.794654 18184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:55.794790 18184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Initialized, Role: FOLLOWER
I20260926 07:30:55.795397 18184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:30:55.795636 18184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:55.795799 18184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:55.795969 18184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:55.798080 18184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:30:55.798559 18184 leader_election.cc:302] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1912f4d4369e46e2be3083c9ee41a2ed; no voters:
I20260926 07:30:55.799113 18184 leader_election.cc:288] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:55.799276 18189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:55.799976 18189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 LEADER]: Becoming Leader. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Running, Role: LEADER
I20260926 07:30:55.800616 18189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:30:55.801287 18184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:55.803488 18191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1912f4d4369e46e2be3083c9ee41a2ed. Latest consensus state: current_term: 1 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:30:55.803699 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:30:55.803936 18191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:55.803987 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:55.805188 18198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:55.810693 18198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:55.819666 18198 catalog_manager.cc:1347] Generated new cluster ID: 31625f54ad624db0a013ae23830512a7
I20260926 07:30:55.819775 18198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:55.861364 18198 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:55.862715 18198 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:55.874465 18198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Generated new TSK 0
I20260926 07:30:55.875602 18198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:30:55.939617 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:56.270417 18210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:56.270759 18210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:56.270926 18210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:56.278374 18210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:56.278541 18210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:56.278719 18210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:30:56.288793 18210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:56.290117 18210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:56.291553 18210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:56.305068 18218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:56.305166 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:56.305097 18215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:56.308558 18210 server_base.cc:1034] running on GCE node
I20260926 07:30:56.309166 18210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:56.310107 18210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:56.311298 18210 hybrid_clock.cc:648] HybridClock initialized: now 1790407856311225 us; error 66 us; skew 500 ppm
I20260926 07:30:56.311615 18210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:56 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407856, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:56.314775 18210 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:30:56.316305 18210 webserver.cc:460] Webserver started at http://127.2.147.193:38757/ using document root <none> and password file <none>
I20260926 07:30:56.317021 18210 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:56.317162 18210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:56.317479 18210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:56.320703 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "6e6bfb57a6684b65958551c3808c9f9d"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "32773f1746303bfc5d9e2d98b3339911"
server_key_iv: "3328c6d971d72b4e85414631836610df"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.321447 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "6e6bfb57a6684b65958551c3808c9f9d"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "32773f1746303bfc5d9e2d98b3339911"
server_key_iv: "3328c6d971d72b4e85414631836610df"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.326649 18210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 07:30:56.330231 18225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.331605 18210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:56.331787 18210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "6e6bfb57a6684b65958551c3808c9f9d"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "32773f1746303bfc5d9e2d98b3339911"
server_key_iv: "3328c6d971d72b4e85414631836610df"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.331979 18210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:56.355357 18210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:56.359463 18210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:56.359802 18210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:56.360806 18210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:56.362151 18210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:56.362224 18210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.362367 18210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:56.362426 18210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.395058 18210 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33063
I20260926 07:30:56.395182 18337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33063 every 8 connection(s)
I20260926 07:30:56.396421 18210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:56.405687 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18210
I20260926 07:30:56.405916 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:56.406709 2639 external_mini_cluster.cc:1428] Setting key 185d153d6c1a11d677b407b29919b33b
Sep 26 07:30:56 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407856, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:56.418284 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:56.398942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43985 (local address 127.2.147.254:34279)
0926 07:30:56.399462 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:56.399466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:56.400293 (+ 827us) server_negotiation.cc:407] Connection header received
0926 07:30:56.402287 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:56.402294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:56.402390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:56.402512 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:56.404394 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.405367 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.407727 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.408142 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.408285 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:56.412163 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:56.412169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:56.412172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:56.412208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:56.414939 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:56.415837 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:56.415842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:56.415844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:56.415969 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:56.416438 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:56.416442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:56.416444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:56.416718 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:56.416884 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:56.417662 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 07:30:56.417910 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":311,"threads_started":1}
I20260926 07:30:56.419574 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:56.399443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:43985)
0926 07:30:56.400139 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:56.400176 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:56.402012 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:56.402759 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:56.402767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:56.403359 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:56.404191 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:56.404215 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.405530 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.405537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:56.407580 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:56.407596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.408373 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.408383 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:56.408712 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:56.409423 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:56.409447 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:56.411872 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:56.415112 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:56.415122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:56.415141 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:56.415689 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:56.416086 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:56.416091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:56.416095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:56.416305 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:56.416866 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:56.416875 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:56.417261 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 07:30:56.417549 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 07:30:56.418153 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260926 07:30:56.421411 18338 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:30:56.421746 18338 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:56.422497 18338 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
I20260926 07:30:56.424698 18149 ts_manager.cc:194] Registered new tserver with Master: 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063)
I20260926 07:30:56.426184 18149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:43985
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:30:56.458333 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:56.794016 18344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:56.794354 18344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:56.794528 18344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:56.801895 18344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:56.802052 18344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:56.802237 18344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:30:56.812077 18344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:56.813357 18344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:56.814667 18344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:56.827702 18352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:56.827771 18349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:56.827749 18350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:56.828563 18344 server_base.cc:1034] running on GCE node
I20260926 07:30:56.829161 18344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:56.830149 18344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:56.831308 18344 hybrid_clock.cc:648] HybridClock initialized: now 1790407856831263 us; error 35 us; skew 500 ppm
I20260926 07:30:56.831629 18344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:56 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407856, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:56.834739 18344 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:30:56.836236 18344 webserver.cc:460] Webserver started at http://127.2.147.194:41401/ using document root <none> and password file <none>
I20260926 07:30:56.836932 18344 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:56.837078 18344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:56.837400 18344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:56.840350 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "3698853f0c33467d8de9622205f404a3"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "00d113b09e70df02980b8c30dd1b9fa3"
server_key_iv: "212fb987ae95fa2bbaeabc61afcd238f"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.841137 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "3698853f0c33467d8de9622205f404a3"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "00d113b09e70df02980b8c30dd1b9fa3"
server_key_iv: "212fb987ae95fa2bbaeabc61afcd238f"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.846334 18344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:30:56.850010 18359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.851477 18344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:56.851655 18344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "3698853f0c33467d8de9622205f404a3"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "00d113b09e70df02980b8c30dd1b9fa3"
server_key_iv: "212fb987ae95fa2bbaeabc61afcd238f"
server_key_version: "encryptionkey@0"
I20260926 07:30:56.851861 18344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:56.877099 18344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:56.881592 18344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:56.881899 18344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:56.882884 18344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:56.884301 18344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:56.884404 18344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.884548 18344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:56.884619 18344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:56.917685 18344 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37623
I20260926 07:30:56.917737 18471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37623 every 8 connection(s)
I20260926 07:30:56.918987 18344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:56.921044 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18344
I20260926 07:30:56.921236 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:56.921768 2639 external_mini_cluster.cc:1428] Setting key 2afb399ab45af528b221a61af731b589
Sep 26 07:30:56 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407856, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:56.939360 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:56.921775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:37717 (local address 127.2.147.254:34279)
0926 07:30:56.922255 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:56.922262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:56.923398 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 07:30:56.925359 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:56.925365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:56.925445 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:56.925575 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:56.927057 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.927874 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.929516 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.930031 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.930162 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:56.933574 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:56.933583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:56.933592 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:56.933628 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:56.936350 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:56.937053 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:56.937060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:56.937069 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:56.937195 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:56.937613 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:56.937619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:56.937633 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:56.937884 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:56.938040 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:56.938718 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 07:30:56.938949 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260926 07:30:56.940335 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:56.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:37717)
0926 07:30:56.923224 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:56.923283 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:56.925085 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:56.925818 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:56.925827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:56.926332 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:56.926888 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:56.926904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.928028 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.928033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:56.929366 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:56.929377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:56.930256 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:56.930263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:56.930494 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:56.931185 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:56.931260 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:56.933351 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:56.936482 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:56.936490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:56.936507 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:56.936921 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:56.937307 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:56.937311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:56.937315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:56.937499 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:56.938018 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:56.938026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:56.938355 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 07:30:56.938613 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 07:30:56.939153 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":318,"threads_started":1}
I20260926 07:30:56.941884 18472 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:30:56.942246 18472 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:56.943154 18472 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:56.944888 18149 ts_manager.cc:194] Registered new tserver with Master: 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623)
I20260926 07:30:56.945745 18149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:37717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:30:56.974107 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:57.302208 18478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:57.302534 18478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:57.302735 18478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:57.310214 18478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:57.310350 18478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:57.310535 18478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:30:57.320637 18478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:30:57.321928 18478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:57.323297 18478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:57.336548 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:57.336520 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:57.336742 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:57.337677 18478 server_base.cc:1034] running on GCE node
I20260926 07:30:57.338241 18478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:57.339119 18478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:57.340318 18478 hybrid_clock.cc:648] HybridClock initialized: now 1790407857340251 us; error 61 us; skew 500 ppm
I20260926 07:30:57.340595 18478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:30:57 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407857, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:57.343655 18478 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:30:57.345021 18478 webserver.cc:460] Webserver started at http://127.2.147.195:36471/ using document root <none> and password file <none>
I20260926 07:30:57.345644 18478 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:57.345745 18478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:57.345994 18478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:57.348941 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "53b5398b931949958f25eec1d54550e6"
format_stamp: "Formatted at 2026-09-26 07:30:57 on dist-test-slave-1dwd"
server_key: "3cc1df6c190ca6e6b98e699bd114834f"
server_key_iv: "5d72d845bb406decb15bdeb3d3f42d9c"
server_key_version: "encryptionkey@0"
I20260926 07:30:57.349726 18478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "53b5398b931949958f25eec1d54550e6"
format_stamp: "Formatted at 2026-09-26 07:30:57 on dist-test-slave-1dwd"
server_key: "3cc1df6c190ca6e6b98e699bd114834f"
server_key_iv: "5d72d845bb406decb15bdeb3d3f42d9c"
server_key_version: "encryptionkey@0"
I20260926 07:30:57.354990 18478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:30:57.358500 18493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:57.359889 18478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:57.360067 18478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "53b5398b931949958f25eec1d54550e6"
format_stamp: "Formatted at 2026-09-26 07:30:57 on dist-test-slave-1dwd"
server_key: "3cc1df6c190ca6e6b98e699bd114834f"
server_key_iv: "5d72d845bb406decb15bdeb3d3f42d9c"
server_key_version: "encryptionkey@0"
I20260926 07:30:57.360281 18478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:57.375381 18478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:57.380043 18478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:57.380349 18478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:57.381309 18478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:57.382634 18478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:57.382707 18478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:57.382882 18478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:57.382936 18478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:57.416131 18478 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:41689
I20260926 07:30:57.416180 18605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:41689 every 8 connection(s)
I20260926 07:30:57.417399 18478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:57.427237 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18478
I20260926 07:30:57.427436 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:57.428025 2639 external_mini_cluster.cc:1428] Setting key 16ebf54633268ccc93a443b1fb3ea965
I20260926 07:30:57.429574 18338 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
Sep 26 07:30:57 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407857, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:30:57.437546 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.420073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:54253 (local address 127.2.147.254:34279)
0926 07:30:57.420283 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.420290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.421439 (+ 1149us) server_negotiation.cc:407] Connection header received
0926 07:30:57.423148 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.423152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.423279 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.423376 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:57.424809 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.425533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.427058 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.427648 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.427745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.432247 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:57.432256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:57.432262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:57.432337 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:57.434758 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.435534 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.435538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.435540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.435671 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.436034 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.436038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.436040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.436316 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:57.436432 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.437145 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.437315 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:30:57.438520 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.420564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:54253)
0926 07:30:57.421281 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.421317 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.422905 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.423581 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.423599 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.424075 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:57.424635 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.424651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.425666 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.425669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.426916 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.426932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.427878 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.427885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.428190 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.429034 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:57.429065 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:57.431931 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:57.434875 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:57.434882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:57.434894 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:57.435438 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:57.435767 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:57.435770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:57.435773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:57.435947 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:57.436455 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:57.436464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:57.436795 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.437062 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.437437 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":312,"threads_started":1}
I20260926 07:30:57.439936 18606 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:30:57.440187 18606 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:57.440841 18606 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
I20260926 07:30:57.442147 18149 ts_manager.cc:194] Registered new tserver with Master: 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689)
I20260926 07:30:57.443001 18149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:54253
I20260926 07:30:57.444507 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:57.463510 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.448720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.2.147.254:34279)
0926 07:30:57.448870 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.448874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.449387 (+ 513us) server_negotiation.cc:407] Connection header received
0926 07:30:57.449516 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.449519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.449580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.449669 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:57.451009 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.451793 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.453677 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.454416 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.454563 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.457494 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:57.457501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:57.457506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:57.457550 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:57.461118 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.461807 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.461810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.461813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.461933 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.462255 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.462258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.462260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.462547 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:57.462748 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.462962 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.463125 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:30:57.467252 18149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:57.474016 18149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:57.501823 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.485404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41689 (local address 127.0.0.1:38202)
0926 07:30:57.488510 (+ 3106us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.488563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.488786 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.491091 (+ 2305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.491102 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.491165 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:57.491795 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.491817 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.494486 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.494494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.497840 (+ 3346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.497866 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.500738 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.500745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.500967 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.501046 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.501210 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.501432 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2779,"thread_start_us":3687,"threads_started":1}
I20260926 07:30:57.501998 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.485575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.2.147.195:41689)
0926 07:30:57.490008 (+ 4433us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.490019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.490059 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:30:57.490199 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.490211 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.490535 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.490745 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:57.492115 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.494230 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.498089 (+ 3859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.500187 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.500334 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.500589 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.501328 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.501527 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4265,"thread_start_us":5071,"threads_started":1}
I20260926 07:30:57.504033 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.486451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.2.147.194:37623)
0926 07:30:57.488907 (+ 2456us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.488920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.489548 (+ 628us) server_negotiation.cc:407] Connection header received
0926 07:30:57.490445 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.490457 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.490760 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.490951 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:57.492273 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.496194 (+ 3921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.501460 (+ 5266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.502855 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.503036 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.503277 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.503503 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.503711 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":629,"threads_started":1}
I20260926 07:30:57.505110 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.485813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37623 (local address 127.0.0.1:53400)
0926 07:30:57.489425 (+ 3612us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.489448 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.490216 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.491373 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.491380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.491417 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:57.492042 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.492059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.496418 (+ 4359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.496426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.500848 (+ 4422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.501290 (+ 442us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.503064 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.503070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.503280 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.503326 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.504198 (+ 872us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.504350 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3447,"mutex_wait_us":1,"thread_start_us":2181,"threads_started":1}
I20260926 07:30:57.506209 18541 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef (DEFAULT_TABLE table=test-table [id=69fde513465b4b18ad840e58ed476bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:57.507809 18541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36ef0b2d44c35b80a81d390dd0bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:57.509482 18407 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef (DEFAULT_TABLE table=test-table [id=69fde513465b4b18ad840e58ed476bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:57.510579 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.489835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.2.147.193:33063)
0926 07:30:57.500687 (+ 10852us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.500699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.501453 (+ 754us) server_negotiation.cc:407] Connection header received
0926 07:30:57.501790 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.501803 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.502094 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.503275 (+ 1181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:57.503885 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.505568 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.508037 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.509438 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.509600 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.509817 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.510093 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.510269 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10628,"thread_start_us":10859,"threads_started":1}
I20260926 07:30:57.510589 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.489677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:35668)
0926 07:30:57.501346 (+ 11669us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.501365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.501567 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.503362 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.503369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.503403 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:57.503707 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.503718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.505727 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.505735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.507848 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.507863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.509682 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.509688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.509872 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.509903 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.509992 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.510234 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11534,"thread_start_us":11346,"threads_started":1}
I20260926 07:30:57.511713 18407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36ef0b2d44c35b80a81d390dd0bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:57.514703 18273 tablet_service.cc:1467] Processing CreateTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef (DEFAULT_TABLE table=test-table [id=69fde513465b4b18ad840e58ed476bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:57.516134 18273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e36ef0b2d44c35b80a81d390dd0bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:57.519599 18624 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Bootstrap starting.
I20260926 07:30:57.522580 18624 tablet_bootstrap.cc:654] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:57.523655 18624 log.cc:824] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:57.526285 18626 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Bootstrap starting.
I20260926 07:30:57.527099 18624 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: No bootstrap required, opened a new log
I20260926 07:30:57.527395 18624 ts_tablet_manager.cc:1397] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260926 07:30:57.529305 18626 tablet_bootstrap.cc:654] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:57.530623 18626 log.cc:824] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:57.531466 18627 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Bootstrap starting.
I20260926 07:30:57.533691 18626 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: No bootstrap required, opened a new log
I20260926 07:30:57.533953 18626 ts_tablet_manager.cc:1397] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260926 07:30:57.534832 18627 tablet_bootstrap.cc:654] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:57.536113 18627 log.cc:824] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:57.536432 18624 raft_consensus.cc:348] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.536867 18624 raft_consensus.cc:374] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:57.536985 18624 raft_consensus.cc:729] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b5398b931949958f25eec1d54550e6, State: Initialized, Role: FOLLOWER
I20260926 07:30:57.537655 18624 consensus_queue.cc:260] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.538941 18606 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
I20260926 07:30:57.539682 18624 ts_tablet_manager.cc:1428] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260926 07:30:57.540220 18627 tablet_bootstrap.cc:492] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: No bootstrap required, opened a new log
I20260926 07:30:57.540496 18627 ts_tablet_manager.cc:1397] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 07:30:57.544265 18626 raft_consensus.cc:348] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.544626 18626 raft_consensus.cc:374] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:57.544750 18626 raft_consensus.cc:729] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3698853f0c33467d8de9622205f404a3, State: Initialized, Role: FOLLOWER
I20260926 07:30:57.545373 18626 consensus_queue.cc:260] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.546468 18472 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
I20260926 07:30:57.547144 18626 ts_tablet_manager.cc:1428] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260926 07:30:57.548919 18627 raft_consensus.cc:348] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.549361 18627 raft_consensus.cc:374] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:57.549475 18627 raft_consensus.cc:729] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6bfb57a6684b65958551c3808c9f9d, State: Initialized, Role: FOLLOWER
I20260926 07:30:57.550098 18627 consensus_queue.cc:260] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.551726 18627 ts_tablet_manager.cc:1428] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 07:30:57.618167 18630 raft_consensus.cc:482] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:57.618461 18630 raft_consensus.cc:504] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.620334 18630 leader_election.cc:288] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063), 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623)
I20260926 07:30:57.626883 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.620823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:50731 (local address 127.2.147.193:33063)
0926 07:30:57.621058 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.621066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.621212 (+ 146us) server_negotiation.cc:407] Connection header received
0926 07:30:57.621464 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.621473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.621560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.621672 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:57.622157 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.623104 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.624983 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.626094 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.626218 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.626271 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.626600 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.626693 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:57.627044 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.620542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.2.147.195:50731)
0926 07:30:57.621078 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.621102 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.621284 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.621713 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.621721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.621750 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:57.621981 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.621992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.623258 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.623264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.624822 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.624834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.626237 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.626243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.626437 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.626475 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.626577 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.626729 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"spinlock_wait_cycles":1280}
I20260926 07:30:57.628038 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.620928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:46411 (local address 127.2.147.194:37623)
0926 07:30:57.621134 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.621138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.621743 (+ 605us) server_negotiation.cc:407] Connection header received
0926 07:30:57.621960 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.621965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.622046 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.622154 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:57.622720 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.623840 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.625843 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.626946 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.627106 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.627189 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.627658 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.627786 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:30:57.628145 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.620854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37623 (local address 127.2.147.195:46411)
0926 07:30:57.621633 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:57.621652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:57.621812 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:57.622168 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:57.622173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:57.622199 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:57.622535 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.622545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.623996 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.624000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.625680 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:57.625696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.627351 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.627357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:57.627502 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.627524 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:30:57.627642 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:30:57.627768 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"spinlock_wait_cycles":62336,"thread_start_us":426,"threads_started":1}
I20260926 07:30:57.628120 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e36ef0b2d44c35b80a81d390dd0bef" candidate_uuid: "53b5398b931949958f25eec1d54550e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6bfb57a6684b65958551c3808c9f9d" is_pre_election: true
I20260926 07:30:57.628736 18293 raft_consensus.cc:2454] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b5398b931949958f25eec1d54550e6 in term 0.
I20260926 07:30:57.629094 18427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e36ef0b2d44c35b80a81d390dd0bef" candidate_uuid: "53b5398b931949958f25eec1d54550e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3698853f0c33467d8de9622205f404a3" is_pre_election: true
I20260926 07:30:57.629444 18494 leader_election.cc:302] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b5398b931949958f25eec1d54550e6, 6e6bfb57a6684b65958551c3808c9f9d; no voters:
I20260926 07:30:57.629570 18427 raft_consensus.cc:2454] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b5398b931949958f25eec1d54550e6 in term 0.
I20260926 07:30:57.629859 18630 raft_consensus.cc:2781] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:57.630012 18630 raft_consensus.cc:482] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:57.630131 18630 raft_consensus.cc:3037] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:57.632642 18630 raft_consensus.cc:504] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.633241 18630 leader_election.cc:288] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063), 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623)
I20260926 07:30:57.633683 18293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e36ef0b2d44c35b80a81d390dd0bef" candidate_uuid: "53b5398b931949958f25eec1d54550e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6bfb57a6684b65958551c3808c9f9d"
I20260926 07:30:57.633730 18427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e36ef0b2d44c35b80a81d390dd0bef" candidate_uuid: "53b5398b931949958f25eec1d54550e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3698853f0c33467d8de9622205f404a3"
I20260926 07:30:57.633899 18293 raft_consensus.cc:3037] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:57.633921 18427 raft_consensus.cc:3037] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:57.636386 18427 raft_consensus.cc:2454] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b5398b931949958f25eec1d54550e6 in term 1.
I20260926 07:30:57.636405 18293 raft_consensus.cc:2454] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b5398b931949958f25eec1d54550e6 in term 1.
I20260926 07:30:57.636844 18496 leader_election.cc:302] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3698853f0c33467d8de9622205f404a3, 53b5398b931949958f25eec1d54550e6; no voters:
I20260926 07:30:57.637224 18630 raft_consensus.cc:2781] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:57.637645 18630 raft_consensus.cc:686] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 1 LEADER]: Becoming Leader. State: Replica: 53b5398b931949958f25eec1d54550e6, State: Running, Role: LEADER
I20260926 07:30:57.638201 18630 consensus_queue.cc:237] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } }
I20260926 07:30:57.643101 18148 catalog_manager.cc:5495] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b5398b931949958f25eec1d54550e6 (127.2.147.195). New cstate: current_term: 1 leader_uuid: "53b5398b931949958f25eec1d54550e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } health_report { overall_health: UNKNOWN } } }
W20260926 07:30:57.648900 18339 tablet.cc:2369] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:30:57.669005 18607 tablet.cc:2369] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:30:57.671247 18473 tablet.cc:2369] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:57.690408 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.684914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.2.147.195:41689)
0926 07:30:57.685125 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.685131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.685151 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:57.685335 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.685339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.685417 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.685568 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:57.685936 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.686657 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.688198 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.688608 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.688750 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.689256 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:57.689806 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:57.689996 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.690047 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.690135 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:30:57.698904 18427 raft_consensus.cc:1261] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Refusing update from remote peer 53b5398b931949958f25eec1d54550e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:57.698904 18293 raft_consensus.cc:1261] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Refusing update from remote peer 53b5398b931949958f25eec1d54550e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:57.699890 18630 consensus_queue.cc:1035] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:57.700249 18634 consensus_queue.cc:1035] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:57.712276 18637 mvcc.cc:205] Tried to move back new op lower bound from 7333510585123950592 to 7333510584890437632. Current Snapshot: MvccSnapshot[applied={T|T < 7333510585123950592}]
I20260926 07:30:57.713569 18639 mvcc.cc:205] Tried to move back new op lower bound from 7333510585123950592 to 7333510584890437632. Current Snapshot: MvccSnapshot[applied={T|T < 7333510585123950592}]
I20260926 07:30:57.716542 18638 mvcc.cc:205] Tried to move back new op lower bound from 7333510585123950592 to 7333510584890437632. Current Snapshot: MvccSnapshot[applied={T|T < 7333510585123950592}]
I20260926 07:30:57.727180 18149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:57.727536 18149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:57.731618 18149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Sending DeleteTablet for 3 replicas of tablet 66e36ef0b2d44c35b80a81d390dd0bef
I20260926 07:30:57.732961 18273 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:57 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:35668
I20260926 07:30:57.733023 18407 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:57 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:53400
I20260926 07:30:57.733076 18541 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e36ef0b2d44c35b80a81d390dd0bef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:57 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:38202
I20260926 07:30:57.734194 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18210
I20260926 07:30:57.746253 18647 tablet_replica.cc:330] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: stopping tablet replica
I20260926 07:30:57.747030 18647 raft_consensus.cc:2229] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:57.747928 18647 raft_consensus.cc:2258] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:57.750895 18647 ts_tablet_manager.cc:1905] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:57.755493 18649 tablet_replica.cc:330] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: stopping tablet replica
I20260926 07:30:57.755919 18649 raft_consensus.cc:2229] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:57.756433 18649 raft_consensus.cc:2258] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:57.759608 18648 tablet_replica.cc:330] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: stopping tablet replica
I20260926 07:30:57.760066 18648 raft_consensus.cc:2229] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:57.760793 18648 raft_consensus.cc:2258] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:57.762971 18648 ts_tablet_manager.cc:1905] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:30:57 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:30:57.766907 18647 ts_tablet_manager.cc:1918] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:57.767083 18647 log.cc:1196] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/66e36ef0b2d44c35b80a81d390dd0bef
I20260926 07:30:57.767656 18647 ts_tablet_manager.cc:1939] T 66e36ef0b2d44c35b80a81d390dd0bef P 6e6bfb57a6684b65958551c3808c9f9d: Deleting consensus metadata
I20260926 07:30:57.769661 18649 ts_tablet_manager.cc:1905] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:57.769760 18133 catalog_manager.cc:4841] TS 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063): tablet 66e36ef0b2d44c35b80a81d390dd0bef (table test-table [id=69fde513465b4b18ad840e58ed476bac]) successfully deleted
I20260926 07:30:57.774497 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.735421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.2.147.193:33063)
0926 07:30:57.735644 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.735653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.751854 (+ 16201us) server_negotiation.cc:407] Connection header received
0926 07:30:57.752254 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.752263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.752374 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.752467 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:57.756833 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.757822 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.759996 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.760690 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.760788 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.766482 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:57.766498 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:57.766520 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:57.766573 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:57.770706 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.771770 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.771779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.771783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.771975 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.772474 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.772479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.772483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.772915 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:57.773197 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.773847 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.774110 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:30:57.775888 18333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:57.776439 18648 ts_tablet_manager.cc:1918] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:57.776630 18648 log.cc:1196] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/66e36ef0b2d44c35b80a81d390dd0bef
I20260926 07:30:57.777262 18648 ts_tablet_manager.cc:1939] T 66e36ef0b2d44c35b80a81d390dd0bef P 53b5398b931949958f25eec1d54550e6: Deleting consensus metadata
I20260926 07:30:57.778275 18649 ts_tablet_manager.cc:1918] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:57.778461 18649 log.cc:1196] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/66e36ef0b2d44c35b80a81d390dd0bef
I20260926 07:30:57.779003 18649 ts_tablet_manager.cc:1939] T 66e36ef0b2d44c35b80a81d390dd0bef P 3698853f0c33467d8de9622205f404a3: Deleting consensus metadata
I20260926 07:30:57.779008 18133 catalog_manager.cc:4841] TS 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689): tablet 66e36ef0b2d44c35b80a81d390dd0bef (table test-table [id=69fde513465b4b18ad840e58ed476bac]) successfully deleted
I20260926 07:30:57.781183 18135 catalog_manager.cc:4841] TS 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623): tablet 66e36ef0b2d44c35b80a81d390dd0bef (table test-table [id=69fde513465b4b18ad840e58ed476bac]) successfully deleted
I20260926 07:30:57.875913 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18210
I20260926 07:30:57.901211 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18344
Sep 26 07:30:57 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:30:57.919636 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:57.902217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.2.147.194:37623)
0926 07:30:57.902442 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:57.902446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:57.902466 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:57.902523 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:57.902526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:57.902598 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:57.902716 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:57.904419 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.905263 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.907192 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:57.907658 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:57.907757 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:57.913125 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:57.913141 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:57.913157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:57.913193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:57.916120 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.916918 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.916927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.916930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.917085 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:57.917604 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:57.917609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:57.917611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:57.918146 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:57.918424 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:57.918752 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 07:30:57.919003 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:30:57.920557 18467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:58.009161 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18344
I20260926 07:30:58.033252 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18478
Sep 26 07:30:58 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:30:58.050046 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:58.034058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.2.147.195:41689)
0926 07:30:58.034282 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:58.034288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:58.034308 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:58.034447 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:58.034452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:58.034515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:58.034639 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:58.035987 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:58.036817 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:58.038047 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:58.038527 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:58.038669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:58.043666 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:58.043679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:58.043700 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:58.043743 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:58.046825 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:58.047990 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:58.047997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:58.047999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:58.048169 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:58.048582 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:58.048597 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:58.048599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:58.048940 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:58.049184 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:58.049516 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 07:30:58.049690 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 07:30:58.050777 18601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:58.177273 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18478
I20260926 07:30:58.196792 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18117
I20260926 07:30:58.197923 18179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:30:58.309193 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18117
I20260926 07:31:18.330442 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--webserver_interface=127.2.147.254
--webserver_port=37917
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:31:18.663921 18655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:18.664213 18655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:18.664395 18655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:18.671679 18655 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:31:18.671787 18655 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:31:18.671914 18655 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:31:18.672013 18655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:31:18.672115 18655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:18.672228 18655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:18.672302 18655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:31:18.672392 18655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:31:18.682495 18655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=37917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:18.683916 18655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:18.685375 18655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:18.708442 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:18.708370 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:18.708416 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:18.709466 18655 server_base.cc:1034] running on GCE node
I20260926 07:31:18.710487 18655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:18.712277 18655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:18.713533 18655 hybrid_clock.cc:648] HybridClock initialized: now 1790407878713468 us; error 41 us; skew 500 ppm
I20260926 07:31:18.713935 18655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:18 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407878, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:18.717547 18655 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:31:18.719357 18655 webserver.cc:460] Webserver started at http://127.2.147.254:37917/ using document root <none> and password file <none>
I20260926 07:31:18.720151 18655 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:18.720304 18655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:18.726565 18655 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 07:31:18.729722 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:18.731173 18655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:31:18.731453 18655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "1912f4d4369e46e2be3083c9ee41a2ed"
format_stamp: "Formatted at 2026-09-26 07:30:55 on dist-test-slave-1dwd"
server_key: "c134087d1ca7e793b20a5bfea5c87786"
server_key_iv: "440ecc678df25f074ba13855bc283e03"
server_key_version: "encryptionkey@0"
I20260926 07:31:18.732592 18655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:18.755445 18655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:18.759557 18655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:18.759866 18655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:18.778848 18655 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34279
I20260926 07:31:18.778872 18722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34279 every 8 connection(s)
I20260926 07:31:18.780439 18655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:31:18.784672 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18655
I20260926 07:31:18.785640 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:33063
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=38757
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:31:18.787941 18723 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:31:18.791365 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap starting.
I20260926 07:31:18.817552 18723 log.cc:824] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:18.829478 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:31:18.830044 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap complete.
I20260926 07:31:18.839720 18723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:18.840636 18723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Initialized, Role: FOLLOWER
I20260926 07:31:18.841342 18723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:18.841627 18723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:31:18.841759 18723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:31:18.841941 18723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:31:18.846063 18723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:18.846629 18723 leader_election.cc:302] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1912f4d4369e46e2be3083c9ee41a2ed; no voters:
I20260926 07:31:18.847329 18723 leader_election.cc:288] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:31:18.847498 18727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:31:18.848683 18727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 LEADER]: Becoming Leader. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Running, Role: LEADER
I20260926 07:31:18.849255 18727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:18.849546 18723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:31:18.851982 18729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1912f4d4369e46e2be3083c9ee41a2ed. Latest consensus state: current_term: 2 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:31:18.852480 18729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:18.853133 18728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:31:18.853503 18728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:18.864283 18734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:31:18.872560 18734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:31:18.874392 18734 catalog_manager.cc:1259] Loaded cluster ID: 31625f54ad624db0a013ae23830512a7
I20260926 07:31:18.874531 18734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:31:18.879276 18744 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:31:18.879704 18734 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:31:18.881253 18734 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Loaded TSK: 0
I20260926 07:31:18.902168 18734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Generated new TSK 1
I20260926 07:31:18.902551 18734 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:31:19.175081 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:19.175432 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:19.175607 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:19.183054 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:19.183168 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:19.183355 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:31:19.194206 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:33063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=38757
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:19.195653 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:19.197024 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:19.212411 18751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:19.212625 18753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:19.212400 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:19.213929 18725 server_base.cc:1034] running on GCE node
I20260926 07:31:19.214747 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:19.216061 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:19.217325 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790407879217253 us; error 51 us; skew 500 ppm
I20260926 07:31:19.217726 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:19 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407879, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:19.221901 18725 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:31:19.223938 18725 webserver.cc:460] Webserver started at http://127.2.147.193:38757/ using document root <none> and password file <none>
I20260926 07:31:19.224939 18725 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:19.225122 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:19.232740 18725 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:31:19.236251 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.237991 18725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:31:19.238166 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "6e6bfb57a6684b65958551c3808c9f9d"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "32773f1746303bfc5d9e2d98b3339911"
server_key_iv: "3328c6d971d72b4e85414631836610df"
server_key_version: "encryptionkey@0"
I20260926 07:31:19.239579 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:19.264555 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:19.270116 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:19.270431 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:19.271658 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:19.273342 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:19.273447 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.273550 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:19.273626 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.311589 18725 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33063
I20260926 07:31:19.311663 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33063 every 8 connection(s)
I20260926 07:31:19.313230 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:31:19.314054 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18725
I20260926 07:31:19.314846 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:37623
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=41401
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:31:19 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407879, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:31:19.340298 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:19.316541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:54825)
0926 07:31:19.317464 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:19.317523 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:19.320291 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:19.321711 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:19.321725 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:19.322511 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:19.323175 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:19.323193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.325190 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.325198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:19.327476 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:19.327494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.328494 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.328503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:19.328814 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:19.329567 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:19.329606 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:19.332336 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:19.335968 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:19.335977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:19.335993 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:19.336442 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:19.337029 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:19.337032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:19.337034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:19.337211 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:19.338269 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:19.338275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:19.338602 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 07:31:19.338827 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 07:31:19.339289 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":379,"threads_started":1}
I20260926 07:31:19.341020 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:19.316383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:54825 (local address 127.2.147.254:34279)
0926 07:31:19.317127 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:19.317144 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:19.317663 (+ 519us) server_negotiation.cc:407] Connection header received
0926 07:31:19.320846 (+ 3183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:19.320859 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:19.321197 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:19.321494 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:19.323469 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.325008 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.327628 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.328242 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.328376 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:19.332655 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:19.332663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:19.332679 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:19.332718 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:19.335813 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:19.336636 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:19.336649 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:19.336661 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:19.336899 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:19.337339 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:19.337347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:19.337353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:19.338130 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:19.338365 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:19.339136 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 07:31:19.339655 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":292,"threads_started":1}
I20260926 07:31:19.343772 18873 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:31:19.344091 18873 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:19.344924 18873 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
I20260926 07:31:19.347193 18688 ts_manager.cc:194] Registered new tserver with Master: 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063)
I20260926 07:31:19.348753 18688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:54825
W20260926 07:31:19.650905 18877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:19.651300 18877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:19.651471 18877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:19.658712 18877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:19.658866 18877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:19.659066 18877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:31:19.669986 18877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=41401
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:19.671397 18877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:19.672729 18877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:19.685833 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:19.685796 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:19.685853 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:19.686873 18877 server_base.cc:1034] running on GCE node
I20260926 07:31:19.687568 18877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:19.688480 18877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:19.689678 18877 hybrid_clock.cc:648] HybridClock initialized: now 1790407879689644 us; error 33 us; skew 500 ppm
I20260926 07:31:19.689999 18877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:19 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407879, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:19.693149 18877 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:31:19.694620 18877 webserver.cc:460] Webserver started at http://127.2.147.194:41401/ using document root <none> and password file <none>
I20260926 07:31:19.695349 18877 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:19.695510 18877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:19.700623 18877 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:31:19.703177 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.704571 18877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:31:19.704739 18877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "3698853f0c33467d8de9622205f404a3"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "00d113b09e70df02980b8c30dd1b9fa3"
server_key_iv: "212fb987ae95fa2bbaeabc61afcd238f"
server_key_version: "encryptionkey@0"
I20260926 07:31:19.705681 18877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:19.729012 18877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:19.732940 18877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:19.733215 18877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:19.734185 18877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:19.735719 18877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:19.735849 18877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.735981 18877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:19.736073 18877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:19.769040 18877 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37623
I20260926 07:31:19.769098 19004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37623 every 8 connection(s)
I20260926 07:31:19.770531 18877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:31:19.772322 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 18877
I20260926 07:31:19.773101 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:41689
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=36471
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:31:19 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407879, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:31:19.798715 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:19.773291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:40691 (local address 127.2.147.254:34279)
0926 07:31:19.773514 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:19.773518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:19.774773 (+ 1255us) server_negotiation.cc:407] Connection header received
0926 07:31:19.776694 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:19.776701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:19.776786 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:19.776894 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:19.778568 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.779787 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.782069 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.782540 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.782686 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:19.792432 (+ 9746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:19.792437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:19.792441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:19.792477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:19.795094 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:19.795860 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:19.795866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:19.795870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:19.796007 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:19.796451 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:19.796456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:19.796460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:19.796756 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:19.796910 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:19.797625 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 07:31:19.798043 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:31:19.800179 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:19.773871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:40691)
0926 07:31:19.774615 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:19.774650 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:19.776401 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:19.777142 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:19.777150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:19.777645 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:19.778317 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:19.778344 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.779968 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.779973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:19.781898 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:19.781913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:19.782775 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:19.782782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:19.783021 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:19.789787 (+ 6766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:19.789819 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:19.792145 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:19.795292 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:19.795301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:19.795324 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:19.795736 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:19.796119 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:19.796123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:19.796127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:19.796331 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:19.796884 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:19.796891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:19.797252 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 07:31:19.797539 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 07:31:19.799078 (+ 1539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260926 07:31:19.801641 19005 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:31:19.801959 19005 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:19.802656 19005 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
I20260926 07:31:19.804060 18688 ts_manager.cc:194] Registered new tserver with Master: 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623)
I20260926 07:31:19.804898 18688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:40691
W20260926 07:31:20.109781 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:20.110072 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:20.110169 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:20.117658 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:20.117774 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:20.117890 19008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:31:20.127656 19008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:41689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=36471
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:20.128901 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:20.130200 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:20.143389 19013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:20.143354 19014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:20.143446 19016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:20.144289 19008 server_base.cc:1034] running on GCE node
I20260926 07:31:20.144945 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:20.145934 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:20.147243 19008 hybrid_clock.cc:648] HybridClock initialized: now 1790407880147136 us; error 65 us; skew 500 ppm
I20260926 07:31:20.147578 19008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:20 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407880, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:20.150733 19008 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:31:20.152154 19008 webserver.cc:460] Webserver started at http://127.2.147.195:36471/ using document root <none> and password file <none>
I20260926 07:31:20.152843 19008 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:20.153004 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:20.158090 19008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:31:20.160766 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:20.162047 19008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:31:20.162233 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "53b5398b931949958f25eec1d54550e6"
format_stamp: "Formatted at 2026-09-26 07:30:57 on dist-test-slave-1dwd"
server_key: "3cc1df6c190ca6e6b98e699bd114834f"
server_key_iv: "5d72d845bb406decb15bdeb3d3f42d9c"
server_key_version: "encryptionkey@0"
I20260926 07:31:20.163352 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:20.185668 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:20.189811 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:20.190086 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:20.191105 19008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:20.192447 19008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:20.192533 19008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:20.192687 19008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:20.192781 19008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:20.225960 19008 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:41689
I20260926 07:31:20.226061 19135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:41689 every 8 connection(s)
I20260926 07:31:20.227545 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:31:20.231528 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19008
Sep 26 07:31:20 dist-test-slave-1dwd krb5kdc[18104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407880, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:31:20.248406 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.232497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.2.147.254:34279)
0926 07:31:20.232968 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.232974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.233327 (+ 353us) server_negotiation.cc:407] Connection header received
0926 07:31:20.233472 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.233479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.233559 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.233676 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.235249 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.236416 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.238060 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.238750 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.238896 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.242009 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.242019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.242026 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.242070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.245228 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.246200 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.246204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.246206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.246389 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.246793 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.246796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.246798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.247117 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.247513 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.247778 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.248012 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260926 07:31:20.250994 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.230215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:48817 (local address 127.2.147.254:34279)
0926 07:31:20.230398 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.230403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.231717 (+ 1314us) server_negotiation.cc:407] Connection header received
0926 07:31:20.234104 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.234109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.234177 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.234261 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.235983 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.237013 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.239388 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.239972 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.240105 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.243710 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.243715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.243718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.243758 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.247055 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.248035 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.248040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.248044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.248201 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.248762 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.248767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.248769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.249102 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.249240 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.250175 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.250372 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:31:20.251910 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.230661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:48817)
0926 07:31:20.231528 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.231587 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.233862 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.234501 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.234509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.234988 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:20.235771 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.235793 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.237229 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.237233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.239157 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.239173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.240213 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.240218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.240435 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.241119 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:20.241146 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:20.243494 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:20.247190 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:20.247200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:20.247243 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:20.247903 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:20.248345 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:20.248350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:20.248354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:20.248632 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:20.249236 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:20.249246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:20.249699 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.250058 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.250615 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":290,"threads_started":1}
I20260926 07:31:20.253104 19136 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34279
I20260926 07:31:20.253407 19136 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:20.254097 19136 heartbeater.cc:511] Master 127.2.147.254:34279 requested a full tablet report, sending...
I20260926 07:31:20.255463 18688 ts_manager.cc:194] Registered new tserver with Master: 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689)
I20260926 07:31:20.256270 18688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:48817
I20260926 07:31:20.263127 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:31:20.269677 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.264016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.2.147.254:34279)
0926 07:31:20.264151 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.264155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.264620 (+ 465us) server_negotiation.cc:407] Connection header received
0926 07:31:20.264869 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.264872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.265006 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.265150 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:31:20.265621 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.266356 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.267830 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.268237 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.268360 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.268835 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:31:20.269033 (+ 198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:31:20.269411 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46760: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 07:31:20.269771 18875 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46760: token expired
W20260926 07:31:20.270243 18609 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.147.254:34279)
I20260926 07:31:20.282759 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.270800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.2.147.254:34279)
0926 07:31:20.270950 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.270954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.270968 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:31:20.271004 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.271007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.271072 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.271150 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.272688 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.273372 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.274659 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.275025 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.275111 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.277829 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.277834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.277837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.277867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.280320 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.280947 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.280950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.280953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.281062 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.281438 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.281441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.281443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.281746 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.281923 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.282211 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.282383 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:31:20.286147 18688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:31:20.293666 18688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:31:20.328420 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.305550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:37623 (local address 127.0.0.1:45828)
0926 07:31:20.318855 (+ 13305us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.318901 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.319108 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.319920 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.319931 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.319968 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:20.320432 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.320449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.323320 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.323325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.325489 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.325505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.327408 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.327415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.327623 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.327685 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.327836 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.328046 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13023,"thread_start_us":14560,"threads_started":1}
I20260926 07:31:20.328431 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.305696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.2.147.194:37623)
0926 07:31:20.317466 (+ 11770us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.317476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.319015 (+ 1539us) server_negotiation.cc:407] Connection header received
0926 07:31:20.319375 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.319384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.319664 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.320661 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:20.320962 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.323107 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.325659 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.327160 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.327323 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.327534 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.327899 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.328075 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11589,"thread_start_us":13867,"threads_started":1}
I20260926 07:31:20.329891 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.306341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.2.147.195:41689)
0926 07:31:20.318205 (+ 11864us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.318218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.320314 (+ 2096us) server_negotiation.cc:407] Connection header received
0926 07:31:20.320640 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.320648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.320889 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.321102 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:20.321716 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.323832 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.326713 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.328087 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.328217 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.328448 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.329361 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.329552 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11608,"thread_start_us":12985,"threads_started":1}
I20260926 07:31:20.330612 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.305718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41689 (local address 127.0.0.1:52376)
0926 07:31:20.320200 (+ 14482us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.320222 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.320442 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.321091 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.321095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.321119 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:20.321488 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.321500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.324065 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.324069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.326541 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.326554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.328980 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.328987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.329159 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.329191 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.329297 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.329446 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14313,"thread_start_us":13741,"threads_started":1}
I20260926 07:31:20.332851 18940 tablet_service.cc:1467] Processing CreateTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 (DEFAULT_TABLE table=test-table [id=e99431398d6c405f9bc0b04f92df2f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:20.333772 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.321495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36060)
0926 07:31:20.326111 (+ 4616us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.326134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.326264 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.326888 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.326892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.326916 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:20.327347 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.327359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.329154 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.329161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.331192 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.331235 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.333046 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.333052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.333212 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.333243 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.333332 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.333466 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4472,"thread_start_us":4313,"threads_started":1}
I20260926 07:31:20.334204 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.320655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.2.147.193:33063)
0926 07:31:20.324816 (+ 4161us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.324827 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.326254 (+ 1427us) server_negotiation.cc:407] Connection header received
0926 07:31:20.326428 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.326438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.326697 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.326911 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:20.327549 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.329028 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.331394 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.332804 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.332958 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.333175 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.333421 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.333687 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3990,"thread_start_us":3862,"threads_started":1}
I20260926 07:31:20.334419 18940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d2dbc5ada34435803d8fbbdc62c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:20.333916 19071 tablet_service.cc:1467] Processing CreateTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 (DEFAULT_TABLE table=test-table [id=e99431398d6c405f9bc0b04f92df2f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:20.335243 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d2dbc5ada34435803d8fbbdc62c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:20.337783 18808 tablet_service.cc:1467] Processing CreateTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 (DEFAULT_TABLE table=test-table [id=e99431398d6c405f9bc0b04f92df2f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:20.339114 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d2dbc5ada34435803d8fbbdc62c967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:20.346871 19152 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Bootstrap starting.
I20260926 07:31:20.350011 19152 tablet_bootstrap.cc:654] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:20.351104 19153 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Bootstrap starting.
I20260926 07:31:20.351389 19152 log.cc:824] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:20.351928 18873 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
I20260926 07:31:20.355569 19153 tablet_bootstrap.cc:654] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:20.356276 19155 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Bootstrap starting.
I20260926 07:31:20.357010 19153 log.cc:824] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:20.359510 19153 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: No bootstrap required, opened a new log
I20260926 07:31:20.359654 19152 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: No bootstrap required, opened a new log
I20260926 07:31:20.359817 19153 ts_tablet_manager.cc:1397] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 07:31:20.359898 19155 tablet_bootstrap.cc:654] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:20.359937 19152 ts_tablet_manager.cc:1397] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260926 07:31:20.361136 19155 log.cc:824] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:20.364328 19155 tablet_bootstrap.cc:492] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: No bootstrap required, opened a new log
I20260926 07:31:20.364619 19155 ts_tablet_manager.cc:1397] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:31:20.367956 19153 raft_consensus.cc:348] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.368363 19153 raft_consensus.cc:374] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:20.368463 19153 raft_consensus.cc:729] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b5398b931949958f25eec1d54550e6, State: Initialized, Role: FOLLOWER
I20260926 07:31:20.369102 19153 consensus_queue.cc:260] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.370195 19136 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
I20260926 07:31:20.370221 19152 raft_consensus.cc:348] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.370651 19152 raft_consensus.cc:374] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:20.370755 19152 raft_consensus.cc:729] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3698853f0c33467d8de9622205f404a3, State: Initialized, Role: FOLLOWER
I20260926 07:31:20.370838 19153 ts_tablet_manager.cc:1428] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260926 07:31:20.371356 19152 consensus_queue.cc:260] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.372481 19005 heartbeater.cc:503] Master 127.2.147.254:34279 was elected leader, sending a full tablet report...
I20260926 07:31:20.373122 19152 ts_tablet_manager.cc:1428] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 07:31:20.376132 19155 raft_consensus.cc:348] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.376633 19155 raft_consensus.cc:374] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:20.376724 19155 raft_consensus.cc:729] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6bfb57a6684b65958551c3808c9f9d, State: Initialized, Role: FOLLOWER
I20260926 07:31:20.377293 19155 consensus_queue.cc:260] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.378849 19155 ts_tablet_manager.cc:1428] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 07:31:20.479399 19137 tablet.cc:2369] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:31:20.522558 19006 tablet.cc:2369] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:31:20.565707 18874 tablet.cc:2369] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:31:20.569134 19159 raft_consensus.cc:482] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:31:20.569430 19159 raft_consensus.cc:504] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.571039 19159 leader_election.cc:288] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689), 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063)
I20260926 07:31:20.579263 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.571421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:46121 (local address 127.2.147.195:41689)
0926 07:31:20.571683 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.571690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.572066 (+ 376us) server_negotiation.cc:407] Connection header received
0926 07:31:20.572297 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.572307 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.572403 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.572515 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:20.573156 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.574151 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.576689 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.578008 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.578213 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.578298 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.578839 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.578982 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:31:20.579263 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.572258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:51445 (local address 127.2.147.193:33063)
0926 07:31:20.572472 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.572479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.572791 (+ 312us) server_negotiation.cc:407] Connection header received
0926 07:31:20.572939 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.572942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.572999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.573070 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:20.573608 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.574519 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.576883 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.578128 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.578269 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.578349 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.578839 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.578980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:31:20.579463 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.572070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.2.147.194:51445)
0926 07:31:20.572673 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.572690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.572818 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.573118 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.573122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.573152 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:20.573448 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.573454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.574638 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.574643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.576724 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.576733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.578322 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.578327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.578642 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.578678 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.578774 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.578974 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":112,"thread_start_us":279,"threads_started":1}
I20260926 07:31:20.579869 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.571333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:41689 (local address 127.2.147.194:46121)
0926 07:31:20.571917 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.571933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:20.572116 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:20.572566 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:20.572571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:20.572613 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:20.572995 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.573004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.574283 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.574286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.576510 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:20.576526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.578281 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.578288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:20.578581 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.578677 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 07:31:20.578774 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:31:20.578974 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":123,"thread_start_us":304,"threads_started":1}
I20260926 07:31:20.580495 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d2dbc5ada34435803d8fbbdc62c967" candidate_uuid: "3698853f0c33467d8de9622205f404a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6bfb57a6684b65958551c3808c9f9d" is_pre_election: true
I20260926 07:31:20.580811 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d2dbc5ada34435803d8fbbdc62c967" candidate_uuid: "3698853f0c33467d8de9622205f404a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b5398b931949958f25eec1d54550e6" is_pre_election: true
I20260926 07:31:20.581121 18828 raft_consensus.cc:2454] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3698853f0c33467d8de9622205f404a3 in term 0.
I20260926 07:31:20.581331 19091 raft_consensus.cc:2454] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3698853f0c33467d8de9622205f404a3 in term 0.
I20260926 07:31:20.581872 18893 leader_election.cc:302] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3698853f0c33467d8de9622205f404a3, 6e6bfb57a6684b65958551c3808c9f9d; no voters:
I20260926 07:31:20.582255 19159 raft_consensus.cc:2781] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:31:20.582397 19159 raft_consensus.cc:482] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:31:20.582532 19159 raft_consensus.cc:3037] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:20.584764 19159 raft_consensus.cc:504] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.585345 19159 leader_election.cc:288] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [CANDIDATE]: Term 1 election: Requested vote from peers 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689), 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063)
I20260926 07:31:20.585816 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d2dbc5ada34435803d8fbbdc62c967" candidate_uuid: "3698853f0c33467d8de9622205f404a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b5398b931949958f25eec1d54550e6"
I20260926 07:31:20.585891 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d2dbc5ada34435803d8fbbdc62c967" candidate_uuid: "3698853f0c33467d8de9622205f404a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6bfb57a6684b65958551c3808c9f9d"
I20260926 07:31:20.586019 19091 raft_consensus.cc:3037] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:20.586148 18828 raft_consensus.cc:3037] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:20.588330 19091 raft_consensus.cc:2454] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3698853f0c33467d8de9622205f404a3 in term 1.
I20260926 07:31:20.588474 18828 raft_consensus.cc:2454] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3698853f0c33467d8de9622205f404a3 in term 1.
I20260926 07:31:20.588749 18893 leader_election.cc:302] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3698853f0c33467d8de9622205f404a3, 53b5398b931949958f25eec1d54550e6; no voters:
I20260926 07:31:20.589043 19159 raft_consensus.cc:2781] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:31:20.589478 19159 raft_consensus.cc:686] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3698853f0c33467d8de9622205f404a3, State: Running, Role: LEADER
I20260926 07:31:20.589941 19159 consensus_queue.cc:237] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } }
I20260926 07:31:20.594451 18687 catalog_manager.cc:5495] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3698853f0c33467d8de9622205f404a3 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "3698853f0c33467d8de9622205f404a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3698853f0c33467d8de9622205f404a3" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 37623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260926 07:31:20.619062 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.614154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.2.147.194:37623)
0926 07:31:20.614348 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.614352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.614462 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:31:20.614621 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.614625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.614707 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.614828 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:31:20.615150 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.615890 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.617219 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.617596 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.617683 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.618141 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:31:20.618630 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:31:20.618737 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.618801 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.618861 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:31:20.627315 19091 raft_consensus.cc:1261] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Refusing update from remote peer 3698853f0c33467d8de9622205f404a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:31:20.627315 18828 raft_consensus.cc:1261] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Refusing update from remote peer 3698853f0c33467d8de9622205f404a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:31:20.628320 19163 consensus_queue.cc:1035] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6bfb57a6684b65958551c3808c9f9d" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:31:20.628681 19159 consensus_queue.cc:1035] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b5398b931949958f25eec1d54550e6" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:31:20.640080 19168 mvcc.cc:205] Tried to move back new op lower bound from 7333510679038009344 to 7333510678900293632. Current Snapshot: MvccSnapshot[applied={T|T < 7333510679038009344}]
I20260926 07:31:20.652655 18687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:31:20.652956 18687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:31:20.656580 18687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Sending DeleteTablet for 3 replicas of tablet 99d2dbc5ada34435803d8fbbdc62c967
I20260926 07:31:20.657768 18940 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:20 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:45828
I20260926 07:31:20.657950 19071 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:20 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260926 07:31:20.658160 18808 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d2dbc5ada34435803d8fbbdc62c967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:20 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:36060
I20260926 07:31:20.658908 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18725
I20260926 07:31:20.658933 19176 tablet_replica.cc:330] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: stopping tablet replica
I20260926 07:31:20.660135 19177 tablet_replica.cc:330] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: stopping tablet replica
I20260926 07:31:20.660571 19177 raft_consensus.cc:2229] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:31:20.661113 19177 raft_consensus.cc:2258] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:31:20.661474 19176 raft_consensus.cc:2229] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:31:20.662107 19176 raft_consensus.cc:2258] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:31:20.666815 19176 ts_tablet_manager.cc:1905] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:20.668488 19177 ts_tablet_manager.cc:1905] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:20.673272 19175 tablet_replica.cc:330] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: stopping tablet replica
I20260926 07:31:20.673758 19175 raft_consensus.cc:2229] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:31:20.674460 19175 raft_consensus.cc:2258] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:31:20.676822 19175 ts_tablet_manager.cc:1905] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:20.678761 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.660287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.2.147.193:33063)
0926 07:31:20.660536 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.660545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.660572 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:31:20.660658 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.660671 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.660776 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.660910 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.662804 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.664360 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.666422 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.667452 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.667621 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.671162 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.671180 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.671224 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.671276 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.675504 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.676460 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.676468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.676472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.676642 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.677046 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.677051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.677054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.677517 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.677818 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.678140 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.678357 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 07:31:20.679935 18868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:20.684397 19175 ts_tablet_manager.cc:1918] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:31:20.684592 19175 log.cc:1196] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/99d2dbc5ada34435803d8fbbdc62c967
I20260926 07:31:20.685177 19175 ts_tablet_manager.cc:1939] T 99d2dbc5ada34435803d8fbbdc62c967 P 3698853f0c33467d8de9622205f404a3: Deleting consensus metadata
I20260926 07:31:20.686496 19176 ts_tablet_manager.cc:1918] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:31:20.686671 19176 log.cc:1196] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/99d2dbc5ada34435803d8fbbdc62c967
I20260926 07:31:20.687110 18674 catalog_manager.cc:4841] TS 3698853f0c33467d8de9622205f404a3 (127.2.147.194:37623): tablet 99d2dbc5ada34435803d8fbbdc62c967 (table test-table [id=e99431398d6c405f9bc0b04f92df2f3d]) successfully deleted
I20260926 07:31:20.687294 19176 ts_tablet_manager.cc:1939] T 99d2dbc5ada34435803d8fbbdc62c967 P 53b5398b931949958f25eec1d54550e6: Deleting consensus metadata
I20260926 07:31:20.689163 18672 catalog_manager.cc:4841] TS 53b5398b931949958f25eec1d54550e6 (127.2.147.195:41689): tablet 99d2dbc5ada34435803d8fbbdc62c967 (table test-table [id=e99431398d6c405f9bc0b04f92df2f3d]) successfully deleted
I20260926 07:31:20.774202 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18725
W20260926 07:31:20.797254 18672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.147.193:33063 (error 108)
I20260926 07:31:20.803769 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18877
I20260926 07:31:20.807370 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.806859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36068)
0926 07:31:20.807036 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.807170 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:31:20.807886 18672 catalog_manager.cc:4584] TS 6e6bfb57a6684b65958551c3808c9f9d (127.2.147.193:33063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99d2dbc5ada34435803d8fbbdc62c967: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
I20260926 07:31:20.823388 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.805378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.2.147.194:37623)
0926 07:31:20.805541 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.805546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.805564 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:31:20.805624 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.805629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.805723 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.805820 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.807741 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.808916 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.810380 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.810859 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.811011 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.817438 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.817450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.817469 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.817505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.820552 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.821294 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.821308 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.821310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.821441 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.821819 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.821824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.821827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.822173 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.822534 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.822766 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.822974 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:31:20.824183 19000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:20.861730 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.861241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36074)
0926 07:31:20.861468 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.861581 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 07:31:20.862911 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.862480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36076)
0926 07:31:20.862673 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.862761 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 07:31:20.923537 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.922934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36080)
0926 07:31:20.923178 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.923324 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:31:20.924679 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.924310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36094)
0926 07:31:20.924456 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:20.924537 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 07:31:20.938342 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18877
I20260926 07:31:20.957158 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19008
I20260926 07:31:20.972169 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:20.957953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.2.147.195:41689)
0926 07:31:20.958177 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:20.958183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:20.958204 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:31:20.958409 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:20.958415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:20.958513 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:20.958636 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:20.960083 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.960890 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.962425 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:20.962953 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:20.963111 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:20.966044 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:20.966059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:20.966076 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:20.966111 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:20.969067 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.969866 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.969873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.969876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.970044 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:20.970472 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:20.970482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:20.970484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:20.970833 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:20.971153 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:20.971548 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 07:31:20.971790 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:31:20.972812 19131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:21.005560 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:21.004978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36106)
0926 07:31:21.005239 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:21.005355 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 07:31:21.006714 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:21.006294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36116)
0926 07:31:21.006506 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:21.006597 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 07:31:21.062458 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19008
I20260926 07:31:21.079382 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 18655
I20260926 07:31:21.080595 18718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:21.202469 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 18655
I20260926 07:31:21.202533 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:21.201878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33063 (local address 127.0.0.1:36126)
0926 07:31:21.202242 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:21.202344 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.193:33063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260926 07:31:41.224884 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--webserver_interface=127.2.147.254
--webserver_port=37917
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:31:41.563494 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:41.563784 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:41.563941 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:41.571616 19183 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:31:41.571727 19183 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:31:41.571763 19183 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:31:41.571882 19183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:31:41.571972 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:41.572083 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:41.572176 19183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:31:41.572263 19183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:31:41.582538 19183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=37917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:41.583945 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:41.585322 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:41.597759 19188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:41.597733 19189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:41.597693 19191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:41.598523 19183 server_base.cc:1034] running on GCE node
I20260926 07:31:41.599347 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:41.600697 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:41.601909 19183 hybrid_clock.cc:648] HybridClock initialized: now 1790407901601827 us; error 59 us; skew 500 ppm
I20260926 07:31:41.602257 19183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:41 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407901, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:41.614019 19183 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:31:41.615569 19183 webserver.cc:460] Webserver started at http://127.2.147.254:37917/ using document root <none> and password file <none>
I20260926 07:31:41.616272 19183 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:41.616367 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:41.621977 19183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:31:41.624959 19198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:41.626410 19183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:31:41.626579 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "1912f4d4369e46e2be3083c9ee41a2ed"
format_stamp: "Formatted at 2026-09-26 07:30:55 on dist-test-slave-1dwd"
server_key: "c134087d1ca7e793b20a5bfea5c87786"
server_key_iv: "440ecc678df25f074ba13855bc283e03"
server_key_version: "encryptionkey@0"
I20260926 07:31:41.627635 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:41.644589 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:41.649533 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:41.649784 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:41.668792 19183 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34279
I20260926 07:31:41.668826 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34279 every 8 connection(s)
I20260926 07:31:41.670378 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:31:41.677631 19250 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:31:41.679560 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19183
I20260926 07:31:41.680420 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:33063
--local_ip_for_outbound_sockets=127.2.147.193
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=38757
--webserver_interface=127.2.147.193
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:31:41.681137 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap starting.
I20260926 07:31:41.701783 19250 log.cc:824] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:41.717399 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:31:41.717959 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Bootstrap complete.
I20260926 07:31:41.726924 19250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:41.727994 19250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Initialized, Role: FOLLOWER
I20260926 07:31:41.728659 19250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:41.728916 19250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:31:41.729045 19250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:31:41.729226 19250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:31:41.733300 19250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:41.733783 19250 leader_election.cc:302] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1912f4d4369e46e2be3083c9ee41a2ed; no voters:
I20260926 07:31:41.734463 19250 leader_election.cc:288] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:31:41.734602 19254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:31:41.736861 19250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:31:41.739488 19254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [term 3 LEADER]: Becoming Leader. State: Replica: 1912f4d4369e46e2be3083c9ee41a2ed, State: Running, Role: LEADER
I20260926 07:31:41.740185 19254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } }
I20260926 07:31:41.744824 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:31:41.745225 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:41.745873 19256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1912f4d4369e46e2be3083c9ee41a2ed. Latest consensus state: current_term: 3 leader_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1912f4d4369e46e2be3083c9ee41a2ed" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34279 } } }
I20260926 07:31:41.746220 19256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:41.748062 19262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:31:41.764400 19262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:31:41.767100 19262 catalog_manager.cc:1259] Loaded cluster ID: 31625f54ad624db0a013ae23830512a7
I20260926 07:31:41.767273 19262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:31:41.772359 19262 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:31:41.774516 19262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Loaded TSK: 0
I20260926 07:31:41.774688 19262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1912f4d4369e46e2be3083c9ee41a2ed: Loaded TSK: 1
F20260926 07:31:41.776149 19262 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd1dabee9ad google::LogMessage::Fail() at ??:0
@ 0x7fd1dabf188a google::LogMessage::SendToLog() at ??:0
@ 0x7fd1dabee4de google::LogMessage::Flush() at ??:0
@ 0x7fd1dabefa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd1de9527d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd1de94173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd1f061d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd1f061c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd1f06a82a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd1f06a80c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd1f06f25e9 std::function<>::operator()() at ??:0
@ 0x7fd1f0620cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd1f061ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd1f06a6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd1f06a6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd1dd83a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd1dd83d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd1dd83d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd1dd818eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd1e0af26db start_thread at ??:0
@ 0x7fd1d9a8861f clone at ??:0
W20260926 07:31:42.094898 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:42.095275 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:42.095386 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:42.102635 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:42.102754 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:42.102880 19252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:31:42.112752 19252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:33063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=38757
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:42.114073 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:42.115411 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:42.128561 19301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:42.128609 19299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:42.128610 19298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:42.132577 19252 server_base.cc:1034] running on GCE node
I20260926 07:31:42.133136 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:42.134126 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:42.135371 19252 hybrid_clock.cc:648] HybridClock initialized: now 1790407902135305 us; error 53 us; skew 500 ppm
I20260926 07:31:42.135694 19252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:42 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407902, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:42.138779 19252 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:31:42.140373 19252 webserver.cc:460] Webserver started at http://127.2.147.193:38757/ using document root <none> and password file <none>
I20260926 07:31:42.141072 19252 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:42.141213 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:42.146858 19252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 07:31:42.149884 19308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:42.151288 19252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:31:42.151463 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "6e6bfb57a6684b65958551c3808c9f9d"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "32773f1746303bfc5d9e2d98b3339911"
server_key_iv: "3328c6d971d72b4e85414631836610df"
server_key_version: "encryptionkey@0"
I20260926 07:31:42.152443 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:42.177305 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:42.181236 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:42.181526 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:42.182507 19252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:42.184357 19315 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 07:31:42.189631 19315 ts_tablet_manager.cc:1852] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 07:31:42.190605 19315 ts_tablet_manager.cc:1905] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:42.198251 19315 ts_tablet_manager.cc:1918] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:31:42.198446 19315 log.cc:1196] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/99d2dbc5ada34435803d8fbbdc62c967
I20260926 07:31:42.198943 19315 ts_tablet_manager.cc:1939] T 99d2dbc5ada34435803d8fbbdc62c967 P 6e6bfb57a6684b65958551c3808c9f9d: Deleting consensus metadata
I20260926 07:31:42.199344 19252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 07:31:42.199483 19252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260926 07:31:42.199630 19252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:42.199729 19252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:42.233242 19252 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33063
I20260926 07:31:42.233294 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33063 every 8 connection(s)
I20260926 07:31:42.234634 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:31:42.236307 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19252
I20260926 07:31:42.237089 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:37623
--local_ip_for_outbound_sockets=127.2.147.194
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=41401
--webserver_interface=127.2.147.194
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:31:42.240514 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.238160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:42461)
0926 07:31:42.239052 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.239357 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":335,"threads_started":1}
W20260926 07:31:42.241232 19422 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34279 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:34279: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
I20260926 07:31:42.242236 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.241787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:41659)
0926 07:31:42.241982 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.242082 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:31:42.243386 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.242925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:49547)
0926 07:31:42.243098 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.243222 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 07:31:42.243600 19422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:31:42.574510 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:42.574805 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:42.574904 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:42.582247 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:42.582363 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:42.582486 19425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:31:42.592494 19425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=41401
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:42.593811 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:42.595134 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:42.608623 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:42.608593 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:42.608712 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:42.609308 19425 server_base.cc:1034] running on GCE node
I20260926 07:31:42.609977 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:42.610914 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:42.612119 19425 hybrid_clock.cc:648] HybridClock initialized: now 1790407902612032 us; error 75 us; skew 500 ppm
I20260926 07:31:42.612442 19425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:42 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407902, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:42.615661 19425 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:31:42.617033 19425 webserver.cc:460] Webserver started at http://127.2.147.194:41401/ using document root <none> and password file <none>
I20260926 07:31:42.617813 19425 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:42.617960 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:42.623446 19425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:31:42.626677 19440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:42.628178 19425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:31:42.628368 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "3698853f0c33467d8de9622205f404a3"
format_stamp: "Formatted at 2026-09-26 07:30:56 on dist-test-slave-1dwd"
server_key: "00d113b09e70df02980b8c30dd1b9fa3"
server_key_iv: "212fb987ae95fa2bbaeabc61afcd238f"
server_key_version: "encryptionkey@0"
I20260926 07:31:42.629390 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:42.660789 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:42.665392 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:42.665683 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:42.666694 19425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:42.668148 19425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:42.668282 19425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:42.668406 19425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:42.668499 19425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:42.702045 19425 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:37623
I20260926 07:31:42.702133 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:37623 every 8 connection(s)
I20260926 07:31:42.703536 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:31:42.704010 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19425
I20260926 07:31:42.704766 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:41689
--local_ip_for_outbound_sockets=127.2.147.195
--tserver_master_addrs=127.2.147.254:34279
--webserver_port=36471
--webserver_interface=127.2.147.195
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:31:42.709009 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.706655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:48153)
0926 07:31:42.707544 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.707820 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":303,"threads_started":1}
W20260926 07:31:42.709736 19553 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34279 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:34279: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
I20260926 07:31:42.710774 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.710362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:53553)
0926 07:31:42.710547 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.710634 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:31:42.711745 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:42.711363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:46191)
0926 07:31:42.711520 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:42.711609 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:31:42.711975 19553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:31:43.032115 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:43.032395 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:43.032558 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:43.039851 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:43.039983 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:43.040174 19556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:31:43.050006 19556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:41689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=36471
--tserver_master_addrs=127.2.147.254:34279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:31:43.051368 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:43.052680 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:43.065430 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:43.065447 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:43.065703 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:43.066174 19556 server_base.cc:1034] running on GCE node
I20260926 07:31:43.066715 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:43.067690 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:43.068926 19556 hybrid_clock.cc:648] HybridClock initialized: now 1790407903068865 us; error 52 us; skew 500 ppm
I20260926 07:31:43.069247 19556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:31:43 dist-test-slave-1dwd krb5kdc[18104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:43.072340 19556 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:31:43.073812 19556 webserver.cc:460] Webserver started at http://127.2.147.195:36471/ using document root <none> and password file <none>
I20260926 07:31:43.074491 19556 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:43.074636 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:43.080096 19556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:31:43.082732 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:43.084074 19556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:31:43.084244 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "53b5398b931949958f25eec1d54550e6"
format_stamp: "Formatted at 2026-09-26 07:30:57 on dist-test-slave-1dwd"
server_key: "3cc1df6c190ca6e6b98e699bd114834f"
server_key_iv: "5d72d845bb406decb15bdeb3d3f42d9c"
server_key_version: "encryptionkey@0"
I20260926 07:31:43.085206 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:43.114384 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:43.118613 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:43.118904 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:43.119963 19556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:43.121300 19556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:43.121429 19556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:43.121562 19556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:43.121652 19556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:43.155047 19556 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:41689
I20260926 07:31:43.155129 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:41689 every 8 connection(s)
I20260926 07:31:43.156548 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:31:43.159638 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19556
I20260926 07:31:43.161793 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:43.159518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:60527)
0926 07:31:43.160338 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:43.160666 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":313,"threads_started":1}
W20260926 07:31:43.162534 19684 heartbeater.cc:644] Failed to heartbeat to 127.2.147.254:34279 (0 consecutive failures): Network error: Failed to ping master at 127.2.147.254:34279: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
I20260926 07:31:43.163302 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:43.162954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:45825)
0926 07:31:43.163080 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:43.163144 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:31:43.164129 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:43.163843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:48573)
0926 07:31:43.163962 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:43.164031 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:31:43.164402 19684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:31:43.245016 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:43.244140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:37315)
0926 07:31:43.244708 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:43.244800 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":318,"threads_started":1}
I20260926 07:31:43.713280 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:43.712513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:47671)
0926 07:31:43.712962 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:43.713053 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":209,"threads_started":1}
I20260926 07:31:44.165783 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:44.164940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:40479)
0926 07:31:44.165450 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:44.165542 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260926 07:31:44.246869 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:44.246005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:42527)
0926 07:31:44.246582 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:44.246693 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":265,"threads_started":1}
I20260926 07:31:44.714984 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:44.714228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:34105)
0926 07:31:44.714717 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:44.714805 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":262,"threads_started":1}
I20260926 07:31:45.167644 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:45.166829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:43427)
0926 07:31:45.167372 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:45.167477 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":233,"threads_started":1}
I20260926 07:31:45.248658 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:45.247812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:43895)
0926 07:31:45.248310 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:45.248407 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260926 07:31:45.716811 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:45.716055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:44065)
0926 07:31:45.716549 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:45.716636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260926 07:31:46.169334 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:46.168623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:41341)
0926 07:31:46.169077 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:46.169161 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260926 07:31:46.250285 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:46.249587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:47835)
0926 07:31:46.250030 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:46.250118 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":221,"threads_started":1}
I20260926 07:31:46.718515 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:46.717715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:40131)
0926 07:31:46.718235 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:46.718318 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":301,"threads_started":1}
I20260926 07:31:47.170933 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:47.170197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:47277)
0926 07:31:47.170664 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:47.170753 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260926 07:31:47.252125 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:47.251346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:40767)
0926 07:31:47.251851 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:47.251941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
I20260926 07:31:47.720276 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:47.719536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:44447)
0926 07:31:47.719996 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:47.720084 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":229,"threads_started":1}
I20260926 07:31:48.172640 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:48.171921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:43893)
0926 07:31:48.172385 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:48.172476 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260926 07:31:48.253938 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:48.253186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:47951)
0926 07:31:48.253693 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:48.253781 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260926 07:31:48.721916 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:48.721185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:52573)
0926 07:31:48.721650 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:48.721746 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260926 07:31:49.174325 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:49.173604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:57211)
0926 07:31:49.174071 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:49.174166 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260926 07:31:49.255971 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:49.255045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:59959)
0926 07:31:49.255664 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:49.255753 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":299,"threads_started":1}
I20260926 07:31:49.723909 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:49.722937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:35637)
0926 07:31:49.723556 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:49.723670 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260926 07:31:50.176301 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:50.175404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:55403)
0926 07:31:50.175959 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:50.176042 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":278,"threads_started":1}
I20260926 07:31:50.257700 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:50.256891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:59139)
0926 07:31:50.257450 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:50.257546 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":319,"threads_started":1}
I20260926 07:31:50.725513 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:50.724800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:59087)
0926 07:31:50.725252 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:50.725336 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260926 07:31:51.178525 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:51.177551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:58461)
0926 07:31:51.178171 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:51.178285 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":347,"threads_started":1}
I20260926 07:31:51.259599 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:51.258744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:41515)
0926 07:31:51.259305 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:51.259427 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
I20260926 07:31:51.727362 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:51.726484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:43939)
0926 07:31:51.726978 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:51.727101 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260926 07:31:52.180455 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:52.179730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:50757)
0926 07:31:52.180202 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:52.180284 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260926 07:31:52.261348 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:52.260623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:59143)
0926 07:31:52.261097 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:52.261188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260926 07:31:52.729192 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:52.728360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:55673)
0926 07:31:52.728896 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:52.729006 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":319,"threads_started":1}
I20260926 07:31:53.182318 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:53.181569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:39807)
0926 07:31:53.182065 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:53.182154 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
I20260926 07:31:53.263195 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:53.262374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:37689)
0926 07:31:53.262907 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:53.262999 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
I20260926 07:31:53.731102 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:53.730301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:35157)
0926 07:31:53.730802 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:53.730921 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
I20260926 07:31:54.184250 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:54.183404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:36063)
0926 07:31:54.183947 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:54.184058 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260926 07:31:54.265022 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:54.264291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:44655)
0926 07:31:54.264768 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:54.264855 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260926 07:31:54.732965 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:54.732211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:58231)
0926 07:31:54.732664 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:54.732780 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260926 07:31:55.186116 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:55.185301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:36553)
0926 07:31:55.185854 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:55.185955 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":310,"threads_started":1}
I20260926 07:31:55.266660 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:55.265912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:45927)
0926 07:31:55.266376 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:55.266476 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":222,"threads_started":1}
I20260926 07:31:55.734808 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:55.733991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:48193)
0926 07:31:55.734531 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:55.734624 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":298,"threads_started":1}
I20260926 07:31:56.188019 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:56.187166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:58055)
0926 07:31:56.187726 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:56.187824 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":248,"threads_started":1}
I20260926 07:31:56.268471 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:56.267736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:52815)
0926 07:31:56.268188 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:56.268279 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260926 07:31:56.736536 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:56.735790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:46407)
0926 07:31:56.736254 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:56.736349 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":254,"threads_started":1}
I20260926 07:31:57.189581 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:57.188897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:43977)
0926 07:31:57.189320 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:57.189409 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":208,"threads_started":1}
I20260926 07:31:57.271044 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:57.269889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:58877)
0926 07:31:57.270724 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:57.270859 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":359,"threads_started":1}
I20260926 07:31:57.738401 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:57.737663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.194:40371)
0926 07:31:57.738132 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:57.738221 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260926 07:31:58.169840 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19252
I20260926 07:31:58.188149 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:58.171031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.2.147.193:33063)
0926 07:31:58.171659 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:58.171683 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:58.171747 (+ 64us) server_negotiation.cc:407] Connection header received
0926 07:31:58.173716 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:58.173726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:58.174153 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:58.174496 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:58.175940 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.177331 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.178580 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.179327 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.179475 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:58.182003 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:58.182010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:58.182024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:58.182062 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:58.185193 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.185799 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.185805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.185811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.185961 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.186277 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.186280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.186281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.186778 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:58.187002 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:58.187392 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 07:31:58.187807 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
I20260926 07:31:58.189566 19417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:58.193549 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:58.190760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.195:33239)
0926 07:31:58.193277 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:58.193382 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2386,"thread_start_us":2285,"threads_started":1}
I20260926 07:31:58.275484 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:58.274427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34279 (local address 127.2.147.193:39515)
0926 07:31:58.275027 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:58.275182 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.254:34279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":295,"threads_started":1}
I20260926 07:31:58.276172 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19252
I20260926 07:31:58.293352 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19425
I20260926 07:31:58.312037 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:58.294320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.2.147.194:37623)
0926 07:31:58.294828 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:58.294842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:58.294878 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:31:58.296547 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:58.296556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:58.297008 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:58.297369 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:58.298852 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.300232 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.301553 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.302365 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.303270 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:58.306009 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:58.306020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:58.306047 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:58.306085 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:58.308913 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.309544 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.309550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.309557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.309725 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.310058 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.310061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.310063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.310634 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:58.310846 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:58.311300 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 07:31:58.311728 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260926 07:31:58.313401 19548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:58.396626 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19425
I20260926 07:31:58.413889 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19556
I20260926 07:31:58.431412 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:58.414997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.2.147.195:41689)
0926 07:31:58.415695 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:58.415709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:58.415749 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:31:58.417275 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:58.417302 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:58.417733 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:58.418029 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:58.419411 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.420670 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.421915 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:58.422588 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:58.422722 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:58.425426 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:58.425435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:58.425450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:58.425509 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:58.428429 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.429084 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.429091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.429097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.429254 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:58.429563 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:58.429566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:58.429568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:58.430076 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:58.430296 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:58.430652 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 07:31:58.431060 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":386,"threads_started":1}
I20260926 07:31:58.432828 19679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:31:58.516438 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19556
2026-09-26T07:31:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64546 ms)
[----------] 30 tests from SecurityITest (295313 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:31:58 dist-test-slave-1dwd krb5kdc[19742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:31:58 dist-test-slave-1dwd krb5kdc[19742](info): set up 2 sockets
Sep 26 07:31:58 dist-test-slave-1dwd krb5kdc[19742](info): commencing operation
krb5kdc: starting...
W20260926 07:31:59.619349 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:31:59 dist-test-slave-1dwd krb5kdc[19742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:31:59.873057 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37971
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:00.209522 19755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:00.209889 19755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:00.210079 19755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:00.217522 19755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:00.217715 19755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:00.217837 19755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:00.217922 19755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:00.218011 19755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:00.228159 19755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:00.229633 19755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:00.231017 19755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:00.243283 19761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:00.243248 19763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:00.243307 19760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:00.243691 19755 server_base.cc:1034] running on GCE node
I20260926 07:32:00.244673 19755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:00.246203 19755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:00.247406 19755 hybrid_clock.cc:648] HybridClock initialized: now 1790407920247330 us; error 59 us; skew 500 ppm
I20260926 07:32:00.247712 19755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:00 dist-test-slave-1dwd krb5kdc[19742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:00.256871 19755 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:00.258507 19755 webserver.cc:460] Webserver started at http://127.2.147.254:44661/ using document root <none> and password file <none>
I20260926 07:32:00.259289 19755 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:00.259433 19755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:00.259765 19755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:00.262836 19755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "7b4ed91e35f5425696349efce7a9ab3b"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "7a3a4a9ab381347190c8fbb3613d263d"
server_key_iv: "6d71a9bc8544e575a48710fad361fccf"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.263696 19755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "7b4ed91e35f5425696349efce7a9ab3b"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "7a3a4a9ab381347190c8fbb3613d263d"
server_key_iv: "6d71a9bc8544e575a48710fad361fccf"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.269140 19755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:32:00.272936 19770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:00.274582 19755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:00.274798 19755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "7b4ed91e35f5425696349efce7a9ab3b"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "7a3a4a9ab381347190c8fbb3613d263d"
server_key_iv: "6d71a9bc8544e575a48710fad361fccf"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.275025 19755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:00.290236 19755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:00.294452 19755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:00.294751 19755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:00.314272 19755 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:37971
I20260926 07:32:00.314281 19821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:37971 every 8 connection(s)
I20260926 07:32:00.315867 19755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:00.317061 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19755
I20260926 07:32:00.317283 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:00.318086 2639 external_mini_cluster.cc:1428] Setting key 501060b099ab1e5bbae2d1994b170c17
I20260926 07:32:00.321095 19822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:00.332134 19822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: Bootstrap starting.
Sep 26 07:32:00 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:00.335855 19822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:00.336933 19822 log.cc:824] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:00.340401 19822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: No bootstrap required, opened a new log
I20260926 07:32:00.341392 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:00.320324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.2.147.254:37971)
0926 07:32:00.321175 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:00.321193 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:00.321245 (+ 52us) server_negotiation.cc:407] Connection header received
0926 07:32:00.322904 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:00.322932 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:00.323422 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:00.323874 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:00.325385 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.326742 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.328204 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.328907 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.329129 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:00.333577 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:00.333585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:00.333599 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:00.333634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:00.337145 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:00.338026 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:00.338039 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:00.338047 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:00.338256 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:00.338690 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:00.338694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:00.338696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:00.339296 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:00.339532 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:00.340002 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 07:32:00.340367 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":375,"threads_started":1}
I20260926 07:32:00.348325 19822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } }
I20260926 07:32:00.348766 19822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:00.348881 19822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4ed91e35f5425696349efce7a9ab3b, State: Initialized, Role: FOLLOWER
I20260926 07:32:00.349471 19822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } }
I20260926 07:32:00.349720 19822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:00.349864 19822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:00.350046 19822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:00.352411 19822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } }
I20260926 07:32:00.352983 19822 leader_election.cc:302] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4ed91e35f5425696349efce7a9ab3b; no voters:
I20260926 07:32:00.353695 19822 leader_election.cc:288] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:00.353811 19827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:00.354501 19827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [term 1 LEADER]: Becoming Leader. State: Replica: 7b4ed91e35f5425696349efce7a9ab3b, State: Running, Role: LEADER
I20260926 07:32:00.355134 19827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } }
I20260926 07:32:00.356099 19822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:00.358434 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b4ed91e35f5425696349efce7a9ab3b. Latest consensus state: current_term: 1 leader_uuid: "7b4ed91e35f5425696349efce7a9ab3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } } }
I20260926 07:32:00.358834 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:00.359462 19828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b4ed91e35f5425696349efce7a9ab3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4ed91e35f5425696349efce7a9ab3b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 37971 } } }
I20260926 07:32:00.359913 19828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:00.361615 19834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:00.367370 19834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:00.376595 19834 catalog_manager.cc:1347] Generated new cluster ID: 51b52b5d9a394888bd6a9711a5b32ecf
I20260926 07:32:00.376719 19834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:00.389068 19834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:00.391098 19834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:00.400949 19834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: Generated new TSK 0
I20260926 07:32:00.402067 19834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:32:00.471694 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:00.807296 19848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:00.807621 19848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:00.807797 19848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:00.815044 19848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:00.815215 19848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:00.815428 19848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:00.825397 19848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:00.826797 19848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:00.828179 19848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:00.841387 19854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:00.841410 19856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:00.842113 19853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:00.847791 19848 server_base.cc:1034] running on GCE node
I20260926 07:32:00.848450 19848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:00.849414 19848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:00.850677 19848 hybrid_clock.cc:648] HybridClock initialized: now 1790407920850608 us; error 61 us; skew 500 ppm
I20260926 07:32:00.851044 19848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:00 dist-test-slave-1dwd krb5kdc[19742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:00.854350 19848 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:32:00.856086 19848 webserver.cc:460] Webserver started at http://127.2.147.193:35789/ using document root <none> and password file <none>
I20260926 07:32:00.856925 19848 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:00.857410 19848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:00.857841 19848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:00.860954 19848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "66909e98c66c4cf09a9819f3278ede07"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "e8a279e5b532fb8439260fd901928cd1"
server_key_iv: "07744e1c776e644faa7b9ba6657bb8ff"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.861902 19848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "66909e98c66c4cf09a9819f3278ede07"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "e8a279e5b532fb8439260fd901928cd1"
server_key_iv: "07744e1c776e644faa7b9ba6657bb8ff"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.867409 19848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 07:32:00.871325 19863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:00.872759 19848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:00.873046 19848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "66909e98c66c4cf09a9819f3278ede07"
format_stamp: "Formatted at 2026-09-26 07:32:00 on dist-test-slave-1dwd"
server_key: "e8a279e5b532fb8439260fd901928cd1"
server_key_iv: "07744e1c776e644faa7b9ba6657bb8ff"
server_key_version: "encryptionkey@0"
I20260926 07:32:00.873272 19848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:00.911397 19848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:00.915684 19848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:00.915997 19848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:00.916985 19848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:00.918332 19848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:00.918455 19848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:00.918599 19848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:00.918675 19848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:00.951681 19848 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:40119
I20260926 07:32:00.951720 19975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:40119 every 8 connection(s)
I20260926 07:32:00.953029 19848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:00.959635 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19848
I20260926 07:32:00.959830 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:00.960480 2639 external_mini_cluster.cc:1428] Setting key c28853cf9f18d1ae130c25f32bb8a6fb
Sep 26 07:32:00 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:00.975832 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:00.955718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:43305 (local address 127.2.147.254:37971)
0926 07:32:00.956209 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:00.956213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:00.957370 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 07:32:00.959221 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:00.959229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:00.959315 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:00.959457 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:00.961072 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.962078 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.964633 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.965032 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.965136 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:00.969379 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:00.969388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:00.969396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:00.969448 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:00.972534 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:00.973479 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:00.973482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:00.973485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:00.973594 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:00.974080 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:00.974083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:00.974085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:00.974343 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:00.974465 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:00.975380 (+ 915us) server_negotiation.cc:299] Negotiation successful
0926 07:32:00.975573 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260926 07:32:00.977479 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:00.956354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37971 (local address 127.2.147.193:43305)
0926 07:32:00.957205 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:00.957250 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:00.958906 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:00.959685 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:00.959693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:00.960307 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:00.960895 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:00.960916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.962264 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.962269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:00.964475 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:00.964495 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:00.965266 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:00.965275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:00.965619 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:00.966383 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:00.966427 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:00.969120 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:00.972683 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:00.972694 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:00.972716 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:00.973357 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:00.973698 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:00.973705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:00.973712 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:00.973975 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:00.974473 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:00.974483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:00.974949 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 07:32:00.975305 (+ 356us) client_negotiation.cc:240] Negotiation successful
0926 07:32:00.975907 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":307,"threads_started":1}
I20260926 07:32:00.979146 19976 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37971
I20260926 07:32:00.979538 19976 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:00.980329 19976 heartbeater.cc:511] Master 127.2.147.254:37971 requested a full tablet report, sending...
I20260926 07:32:00.982488 19787 ts_manager.cc:194] Registered new tserver with Master: 66909e98c66c4cf09a9819f3278ede07 (127.2.147.193:40119)
I20260926 07:32:00.984218 19787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:43305
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:32:01.018399 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:01.349705 19982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:01.350039 19982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:01.350209 19982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:01.357645 19982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:01.357749 19982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:01.357862 19982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:01.367821 19982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:01.369072 19982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:01.370323 19982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:01.383659 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:01.383847 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:01.383975 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:01.384624 19982 server_base.cc:1034] running on GCE node
I20260926 07:32:01.385219 19982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:01.386137 19982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:01.387318 19982 hybrid_clock.cc:648] HybridClock initialized: now 1790407921387261 us; error 57 us; skew 500 ppm
I20260926 07:32:01.387645 19982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:01 dist-test-slave-1dwd krb5kdc[19742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:01.390681 19982 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:32:01.392191 19982 webserver.cc:460] Webserver started at http://127.2.147.194:44925/ using document root <none> and password file <none>
I20260926 07:32:01.392864 19982 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:01.393013 19982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:01.393330 19982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:01.396257 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "f82ac6ad3c03460f9f18da0217bda157"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "154967f03f810b2232caf8088f14db0f"
server_key_iv: "69395187ce0f055e55046b139254ceba"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.397037 19982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "f82ac6ad3c03460f9f18da0217bda157"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "154967f03f810b2232caf8088f14db0f"
server_key_iv: "69395187ce0f055e55046b139254ceba"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.402259 19982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 07:32:01.405712 19997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.407092 19982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:01.407307 19982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "f82ac6ad3c03460f9f18da0217bda157"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "154967f03f810b2232caf8088f14db0f"
server_key_iv: "69395187ce0f055e55046b139254ceba"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.407524 19982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:01.433739 19982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:01.438112 19982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:01.438437 19982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:01.439532 19982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:01.440944 19982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:01.441020 19982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.441103 19982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:01.441148 19982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.474711 19982 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:38091
I20260926 07:32:01.474792 20109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:38091 every 8 connection(s)
I20260926 07:32:01.476136 19982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:01.482795 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 19982
I20260926 07:32:01.483009 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:01.483495 2639 external_mini_cluster.cc:1428] Setting key 3f634dda15ab210818e0d222a53ef125
Sep 26 07:32:01 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:01.497613 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:01.479102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55577 (local address 127.2.147.254:37971)
0926 07:32:01.479596 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:01.479600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:01.480627 (+ 1027us) server_negotiation.cc:407] Connection header received
0926 07:32:01.482488 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:01.482494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:01.482596 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:01.482719 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:01.484700 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:01.485608 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:01.487193 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:01.487677 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:01.487828 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:01.491369 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:01.491374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:01.491378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:01.491411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:01.494144 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:01.495109 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:01.495113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:01.495115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:01.495320 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:01.495847 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:01.495850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:01.495852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:01.496109 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:01.496217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:01.497146 (+ 929us) server_negotiation.cc:299] Negotiation successful
0926 07:32:01.497341 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":301,"threads_started":1}
I20260926 07:32:01.499140 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:01.479719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37971 (local address 127.2.147.194:55577)
0926 07:32:01.480470 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:01.480508 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:01.482197 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:01.483003 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:01.483015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:01.483769 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:01.484515 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:01.484544 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:01.485760 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:01.485766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:01.487031 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:01.487047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:01.487913 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:01.487922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:01.488181 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:01.488916 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:01.488939 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:01.491086 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:01.494306 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:01.494319 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:01.494341 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:01.494986 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:01.495435 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:01.495443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:01.495449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:01.495732 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:01.496239 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:01.496250 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:01.496733 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 07:32:01.497044 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 07:32:01.497639 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":267,"threads_started":1}
I20260926 07:32:01.500876 20110 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37971
I20260926 07:32:01.501223 20110 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:01.501953 20110 heartbeater.cc:511] Master 127.2.147.254:37971 requested a full tablet report, sending...
I20260926 07:32:01.503710 19787 ts_manager.cc:194] Registered new tserver with Master: f82ac6ad3c03460f9f18da0217bda157 (127.2.147.194:38091)
I20260926 07:32:01.505025 19787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:55577
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:32:01.540867 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:01.872347 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:01.872627 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:01.872722 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:01.880334 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:01.880441 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:01.880553 20116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:01.890715 20116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:37971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:01.892001 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:01.893275 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:01.906251 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:01.906220 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:01.906313 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:01.907274 20116 server_base.cc:1034] running on GCE node
I20260926 07:32:01.907959 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:01.908946 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:01.910179 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790407921910114 us; error 60 us; skew 500 ppm
I20260926 07:32:01.910516 20116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:01 dist-test-slave-1dwd krb5kdc[19742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:01.913821 20116 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:32:01.915410 20116 webserver.cc:460] Webserver started at http://127.2.147.195:44505/ using document root <none> and password file <none>
I20260926 07:32:01.916106 20116 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:01.916252 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:01.916582 20116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:01.919654 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "0234d25e7ced44df9fa7fed110c84442"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "75190d158220e01059fc742fe8ab5178"
server_key_iv: "34caf9b1baf79f0d607decbb38b8bb50"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.920498 20116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "0234d25e7ced44df9fa7fed110c84442"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "75190d158220e01059fc742fe8ab5178"
server_key_iv: "34caf9b1baf79f0d607decbb38b8bb50"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.925756 20116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:32:01.929337 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.930680 20116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:01.930882 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "0234d25e7ced44df9fa7fed110c84442"
format_stamp: "Formatted at 2026-09-26 07:32:01 on dist-test-slave-1dwd"
server_key: "75190d158220e01059fc742fe8ab5178"
server_key_iv: "34caf9b1baf79f0d607decbb38b8bb50"
server_key_version: "encryptionkey@0"
I20260926 07:32:01.931100 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:01.953750 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:01.958480 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:01.958775 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:01.959811 20116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:01.961131 20116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:01.961241 20116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.961380 20116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:01.961453 20116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:01.987170 19976 heartbeater.cc:503] Master 127.2.147.254:37971 was elected leader, sending a full tablet report...
I20260926 07:32:01.995477 20116 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:36431
I20260926 07:32:01.995529 20243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:36431 every 8 connection(s)
I20260926 07:32:01.996795 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:02.004829 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20116
I20260926 07:32:02.005007 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:02.005489 2639 external_mini_cluster.cc:1428] Setting key 5f33273fa80aca3a73d65e05c2817b52
Sep 26 07:32:02 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:02.017634 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:01.999333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:40495 (local address 127.2.147.254:37971)
0926 07:32:01.999727 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:01.999732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.000907 (+ 1175us) server_negotiation.cc:407] Connection header received
0926 07:32:02.002724 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.002727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.002808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.002909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:02.004473 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.005368 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.007653 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.008266 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.008439 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.012047 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:02.012052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:02.012056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:02.012114 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:02.014877 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.015646 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.015650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.015653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.015775 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.016153 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.016156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.016159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.016423 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:02.016532 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.017179 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.017343 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260926 07:32:02.018600 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:01.999919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:37971 (local address 127.2.147.195:40495)
0926 07:32:02.000731 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.000781 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.002489 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.003111 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.003119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.003730 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:02.004300 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.004317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.005527 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.005534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.007498 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.007513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.008550 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.008556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.008842 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.009497 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:02.009519 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:02.011819 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:02.015013 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:02.015020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:02.015034 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:02.015536 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:02.015864 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:02.015867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:02.015869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:02.016058 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:02.016522 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:02.016527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:02.016847 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.017090 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.017462 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":334,"threads_started":1}
I20260926 07:32:02.020134 20244 heartbeater.cc:347] Connected to a master server at 127.2.147.254:37971
I20260926 07:32:02.020483 20244 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:02.021201 20244 heartbeater.cc:511] Master 127.2.147.254:37971 requested a full tablet report, sending...
I20260926 07:32:02.022631 19786 ts_manager.cc:194] Registered new tserver with Master: 0234d25e7ced44df9fa7fed110c84442 (127.2.147.195:36431)
I20260926 07:32:02.023590 19786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:40495
I20260926 07:32:02.033553 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:02.050539 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.038381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.2.147.254:37971)
0926 07:32:02.038642 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.038648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.038806 (+ 158us) server_negotiation.cc:407] Connection header received
0926 07:32:02.038948 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.038953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.039022 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.039104 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:02.040586 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.041353 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.042676 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.043107 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.043241 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.045664 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:02.045668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:02.045672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:02.045700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:02.048141 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.048762 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.048766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.048768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.048883 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.049207 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.049210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.049212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.049514 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:02.049725 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.049994 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.050183 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 07:32:02.054894 19786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:02.062036 19786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:02.085556 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.074256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.2.147.195:36431)
0926 07:32:02.075600 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.075609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.076038 (+ 429us) server_negotiation.cc:407] Connection header received
0926 07:32:02.076377 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.076389 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.076689 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.076860 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:02.078403 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.079829 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.083404 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084320 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.084472 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.084725 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.085015 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.085181 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":1101,"threads_started":1}
I20260926 07:32:02.085963 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.073606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:40119 (local address 127.0.0.1:44922)
0926 07:32:02.074645 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.074693 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.074909 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.077250 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.077259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.077291 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:02.077734 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.077744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.080005 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.080010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.082926 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.082940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084754 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.084762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.085175 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.085243 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.085314 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.085497 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":263,"thread_start_us":971,"threads_started":1}
I20260926 07:32:02.085632 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.073786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36431 (local address 127.0.0.1:58902)
0926 07:32:02.075932 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.075949 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.076087 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.077275 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.077282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.077318 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:02.078181 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.078193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.079968 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.080016 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.083255 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.083268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084514 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.084519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.084723 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.084826 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.084971 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.085180 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"mutex_wait_us":167,"thread_start_us":2674,"threads_started":1}
I20260926 07:32:02.086550 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.073772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.2.147.193:40119)
0926 07:32:02.076560 (+ 2788us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.076568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.076594 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:32:02.076686 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.076696 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.077007 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.077499 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:02.077876 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.079860 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.083075 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084523 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.085710 (+ 1187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.085951 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.086023 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.086199 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2474,"threads_started":1}
I20260926 07:32:02.087024 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.075913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:38091 (local address 127.0.0.1:40488)
0926 07:32:02.077055 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.077072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.077182 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.078551 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.078555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.078578 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:02.079014 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.079024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.080719 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.080725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.082751 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.082763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084876 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.084882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.085063 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.085227 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.085315 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.086366 (+ 1051us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":955,"thread_start_us":898,"threads_started":1}
I20260926 07:32:02.087388 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.075615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.2.147.194:38091)
0926 07:32:02.077746 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.077752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.077772 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:32:02.077874 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.077880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.078110 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.078300 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:02.079184 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.080590 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.082892 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.084393 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.084536 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.084746 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.085445 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.085614 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1967,"thread_start_us":1761,"threads_started":1}
I20260926 07:32:02.090317 19911 tablet_service.cc:1467] Processing CreateTablet for tablet 450bc7d865e446d983406315126f6d6f (DEFAULT_TABLE table=test-table [id=836549842f1c41dd93d98cb03164ce27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:02.090370 20179 tablet_service.cc:1467] Processing CreateTablet for tablet 450bc7d865e446d983406315126f6d6f (DEFAULT_TABLE table=test-table [id=836549842f1c41dd93d98cb03164ce27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:02.090572 20045 tablet_service.cc:1467] Processing CreateTablet for tablet 450bc7d865e446d983406315126f6d6f (DEFAULT_TABLE table=test-table [id=836549842f1c41dd93d98cb03164ce27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:02.091769 19911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450bc7d865e446d983406315126f6d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:02.091815 20045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450bc7d865e446d983406315126f6d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:02.091815 20179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 450bc7d865e446d983406315126f6d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:02.102649 20263 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Bootstrap starting.
I20260926 07:32:02.106350 20264 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Bootstrap starting.
I20260926 07:32:02.107623 20263 tablet_bootstrap.cc:654] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:02.108934 20263 log.cc:824] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:02.109768 20265 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Bootstrap starting.
I20260926 07:32:02.109990 20264 tablet_bootstrap.cc:654] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:02.111516 20264 log.cc:824] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:02.113133 20265 tablet_bootstrap.cc:654] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:02.114234 20265 log.cc:824] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:02.114439 20263 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: No bootstrap required, opened a new log
I20260926 07:32:02.114794 20263 ts_tablet_manager.cc:1397] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 07:32:02.117690 20265 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: No bootstrap required, opened a new log
I20260926 07:32:02.117981 20265 ts_tablet_manager.cc:1397] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 07:32:02.118096 20264 tablet_bootstrap.cc:492] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: No bootstrap required, opened a new log
I20260926 07:32:02.118351 20264 ts_tablet_manager.cc:1397] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 07:32:02.124997 20265 raft_consensus.cc:348] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.125569 20265 raft_consensus.cc:374] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:02.125734 20265 raft_consensus.cc:729] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0234d25e7ced44df9fa7fed110c84442, State: Initialized, Role: FOLLOWER
I20260926 07:32:02.125751 20263 raft_consensus.cc:348] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.126211 20263 raft_consensus.cc:374] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:02.126307 20263 raft_consensus.cc:729] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66909e98c66c4cf09a9819f3278ede07, State: Initialized, Role: FOLLOWER
I20260926 07:32:02.126978 20265 consensus_queue.cc:260] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.127118 20263 consensus_queue.cc:260] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.128197 20244 heartbeater.cc:503] Master 127.2.147.254:37971 was elected leader, sending a full tablet report...
I20260926 07:32:02.129060 20264 raft_consensus.cc:348] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.129441 20263 ts_tablet_manager.cc:1428] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260926 07:32:02.129544 20264 raft_consensus.cc:374] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:02.129631 20264 raft_consensus.cc:729] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82ac6ad3c03460f9f18da0217bda157, State: Initialized, Role: FOLLOWER
I20260926 07:32:02.130654 20265 ts_tablet_manager.cc:1428] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260926 07:32:02.130594 20264 consensus_queue.cc:260] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.131733 20110 heartbeater.cc:503] Master 127.2.147.254:37971 was elected leader, sending a full tablet report...
I20260926 07:32:02.132365 20264 ts_tablet_manager.cc:1428] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260926 07:32:02.205549 19977 tablet.cc:2369] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:32:02.228430 20111 tablet.cc:2369] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:32:02.248435 20245 tablet.cc:2369] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:02.421577 20271 raft_consensus.cc:482] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:02.421911 20271 raft_consensus.cc:504] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.423955 20271 leader_election.cc:288] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66909e98c66c4cf09a9819f3278ede07 (127.2.147.193:40119), 0234d25e7ced44df9fa7fed110c84442 (127.2.147.195:36431)
I20260926 07:32:02.430863 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.424276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:48937 (local address 127.2.147.193:40119)
0926 07:32:02.424503 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.424511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.425142 (+ 631us) server_negotiation.cc:407] Connection header received
0926 07:32:02.425349 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.425357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.425454 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.425524 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:02.426166 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.426939 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.428963 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.429809 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.429934 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.430026 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.430445 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.430591 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:02.430893 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.424478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:40291 (local address 127.2.147.195:36431)
0926 07:32:02.424688 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.424696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.425268 (+ 572us) server_negotiation.cc:407] Connection header received
0926 07:32:02.425447 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.425451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.425538 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.425629 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:02.426328 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.427099 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.429148 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.429927 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.430090 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.430165 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.430533 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.430663 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:32:02.430984 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.424089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:40119 (local address 127.2.147.194:48937)
0926 07:32:02.425025 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.425042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.425174 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.425569 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.425576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.425607 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:02.426006 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.426016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.427088 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.427093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.428792 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.428805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.430042 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.430049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.430267 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.430316 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.430391 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.430539 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":28,"thread_start_us":596,"threads_started":1}
I20260926 07:32:02.431566 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.424195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:36431 (local address 127.2.147.194:40291)
0926 07:32:02.425138 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:02.425157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:02.425314 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:02.425672 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:02.425677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:02.425718 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:02.426169 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.426181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.427287 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.427295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.428992 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:02.429007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.430151 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.430156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:02.430342 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.430390 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 07:32:02.430494 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:32:02.430615 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":88,"spinlock_wait_cycles":26496,"thread_start_us":548,"threads_started":1}
I20260926 07:32:02.432219 19931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450bc7d865e446d983406315126f6d6f" candidate_uuid: "f82ac6ad3c03460f9f18da0217bda157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66909e98c66c4cf09a9819f3278ede07" is_pre_election: true
I20260926 07:32:02.432385 20199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450bc7d865e446d983406315126f6d6f" candidate_uuid: "f82ac6ad3c03460f9f18da0217bda157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0234d25e7ced44df9fa7fed110c84442" is_pre_election: true
I20260926 07:32:02.432766 19931 raft_consensus.cc:2454] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82ac6ad3c03460f9f18da0217bda157 in term 0.
I20260926 07:32:02.432883 20199 raft_consensus.cc:2454] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82ac6ad3c03460f9f18da0217bda157 in term 0.
I20260926 07:32:02.433504 19999 leader_election.cc:302] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0234d25e7ced44df9fa7fed110c84442, 66909e98c66c4cf09a9819f3278ede07, f82ac6ad3c03460f9f18da0217bda157; no voters:
I20260926 07:32:02.433883 20271 raft_consensus.cc:2781] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:02.434024 20271 raft_consensus.cc:482] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:02.434129 20271 raft_consensus.cc:3037] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:02.436610 20271 raft_consensus.cc:504] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.437173 20271 leader_election.cc:288] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [CANDIDATE]: Term 1 election: Requested vote from peers 66909e98c66c4cf09a9819f3278ede07 (127.2.147.193:40119), 0234d25e7ced44df9fa7fed110c84442 (127.2.147.195:36431)
I20260926 07:32:02.437635 20199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450bc7d865e446d983406315126f6d6f" candidate_uuid: "f82ac6ad3c03460f9f18da0217bda157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0234d25e7ced44df9fa7fed110c84442"
I20260926 07:32:02.437630 19931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "450bc7d865e446d983406315126f6d6f" candidate_uuid: "f82ac6ad3c03460f9f18da0217bda157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66909e98c66c4cf09a9819f3278ede07"
I20260926 07:32:02.437834 20199 raft_consensus.cc:3037] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:02.437851 19931 raft_consensus.cc:3037] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:02.440155 20199 raft_consensus.cc:2454] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82ac6ad3c03460f9f18da0217bda157 in term 1.
I20260926 07:32:02.440500 19931 raft_consensus.cc:2454] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82ac6ad3c03460f9f18da0217bda157 in term 1.
I20260926 07:32:02.440675 19999 leader_election.cc:302] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0234d25e7ced44df9fa7fed110c84442, f82ac6ad3c03460f9f18da0217bda157; no voters:
I20260926 07:32:02.441031 20271 raft_consensus.cc:2781] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:02.441576 20271 raft_consensus.cc:686] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 1 LEADER]: Becoming Leader. State: Replica: f82ac6ad3c03460f9f18da0217bda157, State: Running, Role: LEADER
I20260926 07:32:02.442163 20271 consensus_queue.cc:237] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } }
I20260926 07:32:02.446677 19785 catalog_manager.cc:5495] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 reported cstate change: term changed from 0 to 1, leader changed from <none> to f82ac6ad3c03460f9f18da0217bda157 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "f82ac6ad3c03460f9f18da0217bda157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82ac6ad3c03460f9f18da0217bda157" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38091 } health_report { overall_health: HEALTHY } } }
I20260926 07:32:02.542992 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.537570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.2.147.194:38091)
0926 07:32:02.537769 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.537777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.537799 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:32:02.538059 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.538063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.538140 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.538293 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:02.538694 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.539462 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.540895 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.541297 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.541433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.541913 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:02.542416 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:02.542585 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.542635 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.542713 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:32:02.551100 20199 raft_consensus.cc:1261] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 1 FOLLOWER]: Refusing update from remote peer f82ac6ad3c03460f9f18da0217bda157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:02.551389 19931 raft_consensus.cc:1261] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 1 FOLLOWER]: Refusing update from remote peer f82ac6ad3c03460f9f18da0217bda157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:02.552145 20274 consensus_queue.cc:1035] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0234d25e7ced44df9fa7fed110c84442" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:02.552498 20271 consensus_queue.cc:1035] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66909e98c66c4cf09a9819f3278ede07" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:02.563004 20277 mvcc.cc:205] Tried to move back new op lower bound from 7333510850759892992 to 7333510850327699456. Current Snapshot: MvccSnapshot[applied={T|T < 7333510850759892992}]
I20260926 07:32:02.564771 20279 mvcc.cc:205] Tried to move back new op lower bound from 7333510850759892992 to 7333510850327699456. Current Snapshot: MvccSnapshot[applied={T|T < 7333510850759892992}]
I20260926 07:32:02.571179 20278 mvcc.cc:205] Tried to move back new op lower bound from 7333510850759892992 to 7333510850327699456. Current Snapshot: MvccSnapshot[applied={T|T < 7333510850759892992 or (T in {7333510850759892992})}]
I20260926 07:32:02.578395 19785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:02.578680 19785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:02.582369 19785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b4ed91e35f5425696349efce7a9ab3b: Sending DeleteTablet for 3 replicas of tablet 450bc7d865e446d983406315126f6d6f
I20260926 07:32:02.583504 19911 tablet_service.cc:1514] Processing DeleteTablet for tablet 450bc7d865e446d983406315126f6d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:02 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44922
I20260926 07:32:02.583504 20179 tablet_service.cc:1514] Processing DeleteTablet for tablet 450bc7d865e446d983406315126f6d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:02 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:58902
I20260926 07:32:02.583570 20045 tablet_service.cc:1514] Processing DeleteTablet for tablet 450bc7d865e446d983406315126f6d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:02 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:40488
I20260926 07:32:02.587037 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19848
I20260926 07:32:02.596112 20287 tablet_replica.cc:330] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: stopping tablet replica
I20260926 07:32:02.596278 20288 tablet_replica.cc:330] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: stopping tablet replica
I20260926 07:32:02.596727 20287 raft_consensus.cc:2229] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:02.596727 20288 raft_consensus.cc:2229] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:02.597263 20288 raft_consensus.cc:2258] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:02.597263 20287 raft_consensus.cc:2258] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:02.598062 20289 tablet_replica.cc:330] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: stopping tablet replica
I20260926 07:32:02.598472 20289 raft_consensus.cc:2229] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:02.599167 20289 raft_consensus.cc:2258] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:02.599828 20287 ts_tablet_manager.cc:1905] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:02.601382 20288 ts_tablet_manager.cc:1905] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:02.604909 20289 ts_tablet_manager.cc:1905] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:02.607422 20287 ts_tablet_manager.cc:1918] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:02.607661 20287 log.cc:1196] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/450bc7d865e446d983406315126f6d6f
Sep 26 07:32:02 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:32:02.608299 20287 ts_tablet_manager.cc:1939] T 450bc7d865e446d983406315126f6d6f P 0234d25e7ced44df9fa7fed110c84442: Deleting consensus metadata
I20260926 07:32:02.610559 19772 catalog_manager.cc:4841] TS 0234d25e7ced44df9fa7fed110c84442 (127.2.147.195:36431): tablet 450bc7d865e446d983406315126f6d6f (table test-table [id=836549842f1c41dd93d98cb03164ce27]) successfully deleted
I20260926 07:32:02.615506 20288 ts_tablet_manager.cc:1918] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:02.615690 20288 log.cc:1196] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/450bc7d865e446d983406315126f6d6f
I20260926 07:32:02.616263 20288 ts_tablet_manager.cc:1939] T 450bc7d865e446d983406315126f6d6f P 66909e98c66c4cf09a9819f3278ede07: Deleting consensus metadata
I20260926 07:32:02.617071 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.588437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.2.147.193:40119)
0926 07:32:02.588612 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.588619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.591739 (+ 3120us) server_negotiation.cc:407] Connection header received
0926 07:32:02.591997 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.592003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.592085 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.592196 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:02.599234 (+ 7038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.600413 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.602539 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.603254 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.603357 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.609024 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:02.609041 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:02.609059 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:02.609103 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:02.613387 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.614245 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.614255 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.614258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.614424 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.614866 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.614871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.614875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.615388 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:02.615699 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.616148 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.616458 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:32:02.617738 19971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:02.618122 19773 catalog_manager.cc:4841] TS 66909e98c66c4cf09a9819f3278ede07 (127.2.147.193:40119): tablet 450bc7d865e446d983406315126f6d6f (table test-table [id=836549842f1c41dd93d98cb03164ce27]) successfully deleted
I20260926 07:32:02.621098 20289 ts_tablet_manager.cc:1918] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:02.621479 20289 log.cc:1196] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/450bc7d865e446d983406315126f6d6f
I20260926 07:32:02.622421 20289 ts_tablet_manager.cc:1939] T 450bc7d865e446d983406315126f6d6f P f82ac6ad3c03460f9f18da0217bda157: Deleting consensus metadata
I20260926 07:32:02.624188 19771 catalog_manager.cc:4841] TS f82ac6ad3c03460f9f18da0217bda157 (127.2.147.194:38091): tablet 450bc7d865e446d983406315126f6d6f (table test-table [id=836549842f1c41dd93d98cb03164ce27]) successfully deleted
I20260926 07:32:02.760984 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19848
I20260926 07:32:02.779054 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19982
Sep 26 07:32:02 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:32:02.796917 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.780025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.2.147.194:38091)
0926 07:32:02.780233 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.780239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.780262 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:32:02.780479 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.780485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.780585 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.780671 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:02.782294 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.783164 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.785178 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.785617 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.785716 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.790655 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:02.790669 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:02.790685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:02.790725 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:02.793972 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.794719 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.794726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.794729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.794866 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.795277 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.795283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.795285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.795658 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:02.796010 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.796291 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.796537 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:32:02.797646 20105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:02.915982 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19982
I20260926 07:32:02.934764 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20116
Sep 26 07:32:02 dist-test-slave-1dwd krb5kdc[19742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:32:02.950019 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:02.935536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.2.147.195:36431)
0926 07:32:02.936028 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:02.936032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:02.936048 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:32:02.936102 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:02.936106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:02.936170 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:02.936260 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:02.937574 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.938323 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.939682 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:02.940066 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:02.940156 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:02.944568 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:02.944584 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:02.944600 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:02.944641 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:02.947486 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.948147 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.948154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.948156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.948289 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:02.948651 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:02.948654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:02.948656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:02.949004 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:02.949207 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:02.949463 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:32:02.949632 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":289,"threads_started":1}
I20260926 07:32:02.950811 20239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:03.039860 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20116
I20260926 07:32:03.056382 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 19755
I20260926 07:32:03.057497 19817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:03.173110 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 19755
2026-09-26T07:32:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:03Z Disabled control of system clock
I20260926 07:32:03.250991 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:36631
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:03.583649 20299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:03.583945 20299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:03.584030 20299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:03.591701 20299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:03.591832 20299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:03.591882 20299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:03.591939 20299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:03.591984 20299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:03.601392 20299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:36631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:03.602898 20299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:03.605047 20299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:03.616894 20305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:03.616926 20304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:03.617658 20307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:03.618079 20299 server_base.cc:1034] running on GCE node
I20260926 07:32:03.618849 20299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:03.620366 20299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:03.621619 20299 hybrid_clock.cc:648] HybridClock initialized: now 1790407923621577 us; error 40 us; skew 500 ppm
I20260926 07:32:03.621999 20299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:03.624387 20299 webserver.cc:460] Webserver started at http://127.2.147.254:45195/ using document root <none> and password file <none>
I20260926 07:32:03.625118 20299 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:03.625259 20299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:03.625584 20299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:03.628610 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "a14b2226b96a41e99ca237e1619e2852"
format_stamp: "Formatted at 2026-09-26 07:32:03 on dist-test-slave-1dwd"
server_key: "cf32b5804d02d616fcca8f0ce21a9c33"
server_key_iv: "b6300a65ec99e08fdd4f53125ba8ab07"
server_key_version: "encryptionkey@0"
I20260926 07:32:03.629480 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "a14b2226b96a41e99ca237e1619e2852"
format_stamp: "Formatted at 2026-09-26 07:32:03 on dist-test-slave-1dwd"
server_key: "cf32b5804d02d616fcca8f0ce21a9c33"
server_key_iv: "b6300a65ec99e08fdd4f53125ba8ab07"
server_key_version: "encryptionkey@0"
I20260926 07:32:03.634778 20299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:32:03.638262 20313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:03.639698 20299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:03.639895 20299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "a14b2226b96a41e99ca237e1619e2852"
format_stamp: "Formatted at 2026-09-26 07:32:03 on dist-test-slave-1dwd"
server_key: "cf32b5804d02d616fcca8f0ce21a9c33"
server_key_iv: "b6300a65ec99e08fdd4f53125ba8ab07"
server_key_version: "encryptionkey@0"
I20260926 07:32:03.640129 20299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:03.674710 20299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:03.675647 20299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:03.675957 20299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:03.695045 20299 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:36631
I20260926 07:32:03.695075 20364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:36631 every 8 connection(s)
I20260926 07:32:03.696498 20299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:03.700264 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:03.704684 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20299
I20260926 07:32:03.704862 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:03.705412 2639 external_mini_cluster.cc:1428] Setting key e5189faa6728fc3cd6e0a526c830b619
I20260926 07:32:03.712117 20365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: Bootstrap starting.
I20260926 07:32:03.715723 20365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:03.716993 20365 log.cc:824] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:03.719892 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:03.707404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.2.147.254:36631)
0926 07:32:03.708390 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:03.708401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:03.708455 (+ 54us) server_negotiation.cc:407] Connection header received
0926 07:32:03.710539 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:03.710561 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:03.711026 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:03.711420 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:03.712080 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:03.713450 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:03.715354 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:03.716125 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:03.716264 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:03.717554 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:03.717561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:03.717575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:03.718136 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:03.718211 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:03.718235 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:03.718373 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:03.718591 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:32:03.718897 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":358,"threads_started":1}
I20260926 07:32:03.720088 20365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: No bootstrap required, opened a new log
I20260926 07:32:03.728093 20365 raft_consensus.cc:348] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } }
I20260926 07:32:03.728596 20365 raft_consensus.cc:374] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:03.728739 20365 raft_consensus.cc:729] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14b2226b96a41e99ca237e1619e2852, State: Initialized, Role: FOLLOWER
I20260926 07:32:03.729378 20365 consensus_queue.cc:260] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } }
I20260926 07:32:03.729622 20365 raft_consensus.cc:388] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:03.729755 20365 raft_consensus.cc:482] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:03.729936 20365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:03.732203 20365 raft_consensus.cc:504] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } }
I20260926 07:32:03.732729 20365 leader_election.cc:302] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a14b2226b96a41e99ca237e1619e2852; no voters:
I20260926 07:32:03.733282 20365 leader_election.cc:288] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:03.733438 20370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:03.734129 20370 raft_consensus.cc:686] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [term 1 LEADER]: Becoming Leader. State: Replica: a14b2226b96a41e99ca237e1619e2852, State: Running, Role: LEADER
I20260926 07:32:03.734601 20370 consensus_queue.cc:237] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } }
I20260926 07:32:03.735376 20365 sys_catalog.cc:564] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:03.737218 20371 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a14b2226b96a41e99ca237e1619e2852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } } }
I20260926 07:32:03.737826 20372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a14b2226b96a41e99ca237e1619e2852. Latest consensus state: current_term: 1 leader_uuid: "a14b2226b96a41e99ca237e1619e2852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14b2226b96a41e99ca237e1619e2852" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 36631 } } }
I20260926 07:32:03.738482 20371 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:03.738662 20372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:03.740311 20375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:03.745981 20375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:03.754695 20375 catalog_manager.cc:1347] Generated new cluster ID: 0feaac0e950b4275a82b33c206483d2c
I20260926 07:32:03.754812 20375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:03.785593 20375 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:03.786895 20375 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:03.801954 20375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: Generated new TSK 0
I20260926 07:32:03.802839 20375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:03.807181 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--builtin_ntp_servers=127.2.147.212:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:04.154690 20389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:04.154990 20389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:04.155154 20389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:04.163136 20389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:04.163290 20389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:04.163471 20389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:04.172817 20389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:04.174153 20389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:04.176081 20389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:04.189002 20394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:04.189082 20395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:04.189034 20397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:04.192664 20389 server_base.cc:1034] running on GCE node
I20260926 07:32:04.193342 20389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:04.194278 20389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:04.195514 20389 hybrid_clock.cc:648] HybridClock initialized: now 1790407924195462 us; error 50 us; skew 500 ppm
I20260926 07:32:04.195844 20389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:04.198164 20389 webserver.cc:460] Webserver started at http://127.2.147.193:38545/ using document root <none> and password file <none>
I20260926 07:32:04.198938 20389 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:04.199081 20389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:04.199465 20389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:04.202363 20389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "196f56f7c2834f44a14afd495caed7e1"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "761957f02544721dcd2dca17b4fbcd62"
server_key_iv: "de6049e3da5dd6331c9b8c97857232cc"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.203172 20389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "196f56f7c2834f44a14afd495caed7e1"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "761957f02544721dcd2dca17b4fbcd62"
server_key_iv: "de6049e3da5dd6331c9b8c97857232cc"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.208402 20389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:32:04.212039 20403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.213655 20389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:04.213828 20389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "196f56f7c2834f44a14afd495caed7e1"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "761957f02544721dcd2dca17b4fbcd62"
server_key_iv: "de6049e3da5dd6331c9b8c97857232cc"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.214068 20389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:04.247905 20389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:04.248828 20389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:04.249210 20389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:04.250242 20389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:04.251897 20389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:04.252019 20389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.252171 20389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:04.252290 20389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.287334 20389 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:45889
I20260926 07:32:04.287352 20515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:45889 every 8 connection(s)
I20260926 07:32:04.288844 20389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:04.296927 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20389
I20260926 07:32:04.297096 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:04.297623 2639 external_mini_cluster.cc:1428] Setting key 5c337dda0f6e5837e707e03d9ed1e748
I20260926 07:32:04.300529 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--builtin_ntp_servers=127.2.147.212:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:04.302533 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:04.291566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:47001 (local address 127.2.147.254:36631)
0926 07:32:04.292045 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:04.292049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:04.293129 (+ 1080us) server_negotiation.cc:407] Connection header received
0926 07:32:04.294958 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:04.294965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:04.295056 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:04.295180 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:04.296314 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.297424 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.299538 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.299954 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.300098 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:04.301623 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:04.301628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:04.301631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:04.301649 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:04.301706 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:04.301719 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:04.301822 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:04.302067 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:32:04.302175 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":260,"threads_started":1}
I20260926 07:32:04.303653 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:04.292090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:36631 (local address 127.2.147.193:47001)
0926 07:32:04.292976 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:04.293012 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:04.294675 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:04.295450 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:04.295459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:04.295550 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:04.296137 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:04.296158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.297581 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.297586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:04.299355 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:04.299371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.300188 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.300197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:04.300463 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:04.301265 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:04.301267 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:04.301325 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:04.301415 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:04.301821 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:04.301835 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:04.301845 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 07:32:04.302044 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:32:04.302481 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":284,"threads_started":1}
I20260926 07:32:04.305298 20516 heartbeater.cc:347] Connected to a master server at 127.2.147.254:36631
I20260926 07:32:04.305658 20516 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:04.306607 20516 heartbeater.cc:511] Master 127.2.147.254:36631 requested a full tablet report, sending...
I20260926 07:32:04.308830 20330 ts_manager.cc:194] Registered new tserver with Master: 196f56f7c2834f44a14afd495caed7e1 (127.2.147.193:45889)
I20260926 07:32:04.310290 20330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.193:47001
W20260926 07:32:04.627492 20520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:04.627774 20520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:04.627961 20520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:04.635282 20520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:04.635385 20520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:04.635600 20520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:04.644939 20520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:04.646651 20520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:04.648506 20520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:04.661865 20525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:04.661935 20526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:04.662317 20528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:04.662779 20520 server_base.cc:1034] running on GCE node
I20260926 07:32:04.663537 20520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:04.664649 20520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:04.665894 20520 hybrid_clock.cc:648] HybridClock initialized: now 1790407924665846 us; error 61 us; skew 500 ppm
I20260926 07:32:04.666244 20520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:04.668695 20520 webserver.cc:460] Webserver started at http://127.2.147.194:45671/ using document root <none> and password file <none>
I20260926 07:32:04.669459 20520 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:04.669597 20520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:04.669945 20520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:04.672981 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "78f0cca46945493b8fc962ac8d544638"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "496e35999c93033081b2a9b52d6dd040"
server_key_iv: "e55c77b557a4e1e132d3fb14291c0d71"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.673805 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "78f0cca46945493b8fc962ac8d544638"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "496e35999c93033081b2a9b52d6dd040"
server_key_iv: "e55c77b557a4e1e132d3fb14291c0d71"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.679178 20520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:32:04.682853 20534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.684293 20520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:32:04.684478 20520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "78f0cca46945493b8fc962ac8d544638"
format_stamp: "Formatted at 2026-09-26 07:32:04 on dist-test-slave-1dwd"
server_key: "496e35999c93033081b2a9b52d6dd040"
server_key_iv: "e55c77b557a4e1e132d3fb14291c0d71"
server_key_version: "encryptionkey@0"
I20260926 07:32:04.684705 20520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:04.695796 20520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:04.696704 20520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:04.697014 20520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:04.697994 20520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:04.699483 20520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:04.699602 20520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.699741 20520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:04.699828 20520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:04.733439 20520 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:33925
I20260926 07:32:04.733494 20646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:33925 every 8 connection(s)
I20260926 07:32:04.734799 20520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:04.735912 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20520
I20260926 07:32:04.740527 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:04.741082 2639 external_mini_cluster.cc:1428] Setting key 63441fb3b6b9291aab98839f0747fa6a
I20260926 07:32:04.744239 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--builtin_ntp_servers=127.2.147.212:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:04.750216 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:04.738037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:59459 (local address 127.2.147.254:36631)
0926 07:32:04.738261 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:04.738268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:04.739931 (+ 1663us) server_negotiation.cc:407] Connection header received
0926 07:32:04.742358 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:04.742364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:04.742452 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:04.742560 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:04.743866 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.744864 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.747396 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.747788 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.747894 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:04.749496 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:04.749501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:04.749504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:04.749522 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:04.749551 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:04.749561 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:04.749626 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:04.749901 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:32:04.749980 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:04.751756 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:04.738559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:36631 (local address 127.2.147.194:59459)
0926 07:32:04.739721 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:04.739769 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:04.742085 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:04.742742 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:04.742751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:04.742867 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:04.743696 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:04.743721 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.745024 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.745033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:04.747175 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:04.747191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:04.748027 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:04.748036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:04.748354 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:04.749123 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:04.749129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:04.749184 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:04.749293 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:04.749659 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:04.749672 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:04.749684 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:32:04.749854 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 07:32:04.750282 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":310,"threads_started":1}
I20260926 07:32:04.753243 20647 heartbeater.cc:347] Connected to a master server at 127.2.147.254:36631
I20260926 07:32:04.753573 20647 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:04.754492 20647 heartbeater.cc:511] Master 127.2.147.254:36631 requested a full tablet report, sending...
I20260926 07:32:04.756064 20330 ts_manager.cc:194] Registered new tserver with Master: 78f0cca46945493b8fc962ac8d544638 (127.2.147.194:33925)
I20260926 07:32:04.756847 20330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.194:59459
W20260926 07:32:05.091387 20650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:05.091764 20650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:05.091964 20650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:05.099401 20650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:05.099498 20650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:05.099706 20650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:05.111025 20650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:05.112385 20650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:05.114122 20650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:05.127998 20655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:05.127959 20656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:05.128042 20658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:05.129226 20650 server_base.cc:1034] running on GCE node
I20260926 07:32:05.129935 20650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:05.130954 20650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:05.132148 20650 hybrid_clock.cc:648] HybridClock initialized: now 1790407925132073 us; error 71 us; skew 500 ppm
I20260926 07:32:05.132469 20650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:05.134824 20650 webserver.cc:460] Webserver started at http://127.2.147.195:36657/ using document root <none> and password file <none>
I20260926 07:32:05.135634 20650 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:05.135761 20650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:05.136130 20650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:05.138932 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "9c0f68d86da14ce39e0815b885b41e73"
format_stamp: "Formatted at 2026-09-26 07:32:05 on dist-test-slave-1dwd"
server_key: "c4fe3759539b989f6fbeb7510a1d692c"
server_key_iv: "eb53ed91c5e3e2ccd040836e4434015b"
server_key_version: "encryptionkey@0"
I20260926 07:32:05.139709 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "9c0f68d86da14ce39e0815b885b41e73"
format_stamp: "Formatted at 2026-09-26 07:32:05 on dist-test-slave-1dwd"
server_key: "c4fe3759539b989f6fbeb7510a1d692c"
server_key_iv: "eb53ed91c5e3e2ccd040836e4434015b"
server_key_version: "encryptionkey@0"
I20260926 07:32:05.144613 20650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:32:05.147927 20664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:05.149199 20650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:05.149371 20650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "9c0f68d86da14ce39e0815b885b41e73"
format_stamp: "Formatted at 2026-09-26 07:32:05 on dist-test-slave-1dwd"
server_key: "c4fe3759539b989f6fbeb7510a1d692c"
server_key_iv: "eb53ed91c5e3e2ccd040836e4434015b"
server_key_version: "encryptionkey@0"
I20260926 07:32:05.149585 20650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:05.170431 20650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:05.171432 20650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:05.171737 20650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:05.172696 20650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:05.174141 20650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:05.174263 20650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:05.174388 20650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:05.174474 20650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:05.207023 20650 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:37035
I20260926 07:32:05.207109 20776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:37035 every 8 connection(s)
I20260926 07:32:05.208421 20650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:05.211725 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20650
I20260926 07:32:05.211912 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:05.212356 2639 external_mini_cluster.cc:1428] Setting key eed41d7379b1b2b545949d7b20374306
I20260926 07:32:05.221375 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.210990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:47129 (local address 127.2.147.254:36631)
0926 07:32:05.211174 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.211179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.212949 (+ 1770us) server_negotiation.cc:407] Connection header received
0926 07:32:05.215013 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.215020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.215111 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.215196 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:05.216266 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.217008 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.218581 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.219079 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.219179 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.220588 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:05.220593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:05.220596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:05.220615 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:05.220652 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:05.220663 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:05.220742 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.221045 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.221127 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:32:05.222431 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.211672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:36631 (local address 127.2.147.195:47129)
0926 07:32:05.212770 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.212814 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.214721 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.215455 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.215463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.215545 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:05.216084 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.216099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.217143 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.217147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.218458 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.218470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.219318 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.219324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.219580 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.220240 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:05.220244 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:05.220300 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:05.220387 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:05.220768 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:05.220777 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:05.220784 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.220980 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.221293 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":370,"threads_started":1}
I20260926 07:32:05.223912 20777 heartbeater.cc:347] Connected to a master server at 127.2.147.254:36631
I20260926 07:32:05.224246 20777 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:05.225010 20777 heartbeater.cc:511] Master 127.2.147.254:36631 requested a full tablet report, sending...
I20260926 07:32:05.226598 20330 ts_manager.cc:194] Registered new tserver with Master: 9c0f68d86da14ce39e0815b885b41e73 (127.2.147.195:37035)
I20260926 07:32:05.227756 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:05.227804 20330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.195:47129
I20260926 07:32:05.237874 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.232158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.2.147.254:36631)
0926 07:32:05.232319 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.232324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.232731 (+ 407us) server_negotiation.cc:407] Connection header received
0926 07:32:05.232840 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.232844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.232948 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.233048 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:05.233491 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.234177 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.235621 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.236143 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.236239 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.237286 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:05.237291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:05.237294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:05.237312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:05.237360 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:05.237372 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:05.237449 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.237567 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.237638 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:32:05.241868 20330 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:05.249271 20330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:05.272372 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.261653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.2.147.193:45889)
0926 07:32:05.263397 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.263418 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.263454 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:32:05.263575 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.263585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.263872 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.264080 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:05.265181 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.266837 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.269788 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.271141 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.271374 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.271637 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.271853 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.272013 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":1368,"threads_started":1}
I20260926 07:32:05.272439 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.261460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:45889 (local address 127.0.0.1:43302)
0926 07:32:05.262380 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.262442 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.262664 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.264457 (+ 1793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.264464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.264504 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:05.264967 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.264980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.266979 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.266985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.269540 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.269555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.271399 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.271405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.271594 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.271640 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.271801 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.272006 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":50,"thread_start_us":345,"threads_started":1}
I20260926 07:32:05.273531 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.261821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:37035 (local address 127.0.0.1:47150)
0926 07:32:05.264651 (+ 2830us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.264685 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.264826 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.265851 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.265856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.265880 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:05.266326 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.266338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.268166 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.268173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.270313 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.270324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.272453 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.272460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.272682 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.272844 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.272933 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.273147 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2668,"mutex_wait_us":290,"thread_start_us":2526,"threads_started":1}
I20260926 07:32:05.273558 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.263921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33925 (local address 127.0.0.1:39252)
0926 07:32:05.264975 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.264998 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.265116 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.266166 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.266170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.266197 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:05.266584 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.266595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.268268 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.268274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.270633 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.270654 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.272465 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.272471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.272681 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.272841 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.272924 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.273136 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":139,"thread_start_us":774,"threads_started":1}
I20260926 07:32:05.273895 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.262435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.2.147.195:37035)
0926 07:32:05.264194 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.264210 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.264987 (+ 777us) server_negotiation.cc:407] Connection header received
0926 07:32:05.265109 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.265120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.265403 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.265581 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:05.266523 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.268018 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.270504 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.271877 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.272017 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.272288 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.273040 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.273203 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":1440,"threads_started":1}
I20260926 07:32:05.274878 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.264401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.2.147.194:33925)
0926 07:32:05.265587 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.265598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.265627 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:32:05.265718 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.265725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.265945 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.266766 (+ 821us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:05.267042 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.268053 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.270827 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.272242 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.273665 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.274293 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.274360 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.274533 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":1158,"threads_started":1}
I20260926 07:32:05.276773 20451 tablet_service.cc:1467] Processing CreateTablet for tablet e3d470e214ad4346a28f07d8dcc51606 (DEFAULT_TABLE table=test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:05.277412 20712 tablet_service.cc:1467] Processing CreateTablet for tablet e3d470e214ad4346a28f07d8dcc51606 (DEFAULT_TABLE table=test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:05.277740 20582 tablet_service.cc:1467] Processing CreateTablet for tablet e3d470e214ad4346a28f07d8dcc51606 (DEFAULT_TABLE table=test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:05.278260 20451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d470e214ad4346a28f07d8dcc51606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:05.278775 20712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d470e214ad4346a28f07d8dcc51606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:05.278869 20582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d470e214ad4346a28f07d8dcc51606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:05.297241 20795 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Bootstrap starting.
I20260926 07:32:05.299485 20796 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Bootstrap starting.
I20260926 07:32:05.301700 20797 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Bootstrap starting.
I20260926 07:32:05.303395 20796 tablet_bootstrap.cc:654] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:05.303545 20795 tablet_bootstrap.cc:654] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:05.304530 20795 log.cc:824] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:05.304652 20796 log.cc:824] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:05.306466 20797 tablet_bootstrap.cc:654] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:05.307449 20795 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: No bootstrap required, opened a new log
I20260926 07:32:05.307641 20797 log.cc:824] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:05.307710 20795 ts_tablet_manager.cc:1397] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 07:32:05.308598 20796 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: No bootstrap required, opened a new log
I20260926 07:32:05.308851 20796 ts_tablet_manager.cc:1397] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 07:32:05.310587 20797 tablet_bootstrap.cc:492] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: No bootstrap required, opened a new log
I20260926 07:32:05.310850 20797 ts_tablet_manager.cc:1397] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260926 07:32:05.313299 20516 heartbeater.cc:503] Master 127.2.147.254:36631 was elected leader, sending a full tablet report...
I20260926 07:32:05.317584 20796 raft_consensus.cc:348] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.318053 20796 raft_consensus.cc:374] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:05.318182 20796 raft_consensus.cc:729] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196f56f7c2834f44a14afd495caed7e1, State: Initialized, Role: FOLLOWER
I20260926 07:32:05.317950 20795 raft_consensus.cc:348] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.318473 20795 raft_consensus.cc:374] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:05.318617 20795 raft_consensus.cc:729] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f0cca46945493b8fc962ac8d544638, State: Initialized, Role: FOLLOWER
I20260926 07:32:05.318876 20796 consensus_queue.cc:260] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.319289 20795 consensus_queue.cc:260] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.320739 20796 ts_tablet_manager.cc:1428] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 07:32:05.322652 20797 raft_consensus.cc:348] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.323109 20797 raft_consensus.cc:374] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:05.323254 20797 raft_consensus.cc:729] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0f68d86da14ce39e0815b885b41e73, State: Initialized, Role: FOLLOWER
I20260926 07:32:05.323843 20797 consensus_queue.cc:260] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.325055 20777 heartbeater.cc:503] Master 127.2.147.254:36631 was elected leader, sending a full tablet report...
I20260926 07:32:05.325500 20797 ts_tablet_manager.cc:1428] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260926 07:32:05.326565 20647 heartbeater.cc:503] Master 127.2.147.254:36631 was elected leader, sending a full tablet report...
I20260926 07:32:05.327627 20795 ts_tablet_manager.cc:1428] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260926 07:32:05.364137 20801 raft_consensus.cc:482] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:05.364413 20801 raft_consensus.cc:504] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.366092 20801 leader_election.cc:288] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c0f68d86da14ce39e0815b885b41e73 (127.2.147.195:37035), 78f0cca46945493b8fc962ac8d544638 (127.2.147.194:33925)
I20260926 07:32:05.372663 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.366354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:47087 (local address 127.2.147.195:37035)
0926 07:32:05.366579 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.366586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.367237 (+ 651us) server_negotiation.cc:407] Connection header received
0926 07:32:05.367441 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.367450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.367508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.367605 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:05.368216 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.368933 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.370914 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.371730 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.371946 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.372022 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.372303 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.372422 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:32:05.372663 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.366772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:41753 (local address 127.2.147.194:33925)
0926 07:32:05.367026 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.367033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.367359 (+ 326us) server_negotiation.cc:407] Connection header received
0926 07:32:05.367566 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.367571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.367639 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.367739 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:05.368341 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.369021 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.370903 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.371719 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.371903 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.371984 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.372272 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.372394 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:32:05.372785 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.366348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:33925 (local address 127.2.147.193:41753)
0926 07:32:05.367249 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.367268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.367428 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.367761 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.367765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.367805 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:05.368186 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.368195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.369168 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.369173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.370698 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.370711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.371939 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.371945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.372114 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.372137 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.372243 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.372373 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":2,"thread_start_us":689,"threads_started":1}
I20260926 07:32:05.373045 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.366310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:37035 (local address 127.2.147.193:47087)
0926 07:32:05.367092 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:05.367108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:05.367290 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:05.367619 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:05.367623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:05.367646 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:05.368070 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.368079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.369060 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.369068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.370713 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:05.370724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.371998 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.372006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:05.372157 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.372178 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:32:05.372278 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:32:05.372404 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":107,"spinlock_wait_cycles":18304,"thread_start_us":462,"threads_started":1}
I20260926 07:32:05.373840 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d470e214ad4346a28f07d8dcc51606" candidate_uuid: "196f56f7c2834f44a14afd495caed7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f0cca46945493b8fc962ac8d544638" is_pre_election: true
I20260926 07:32:05.374032 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d470e214ad4346a28f07d8dcc51606" candidate_uuid: "196f56f7c2834f44a14afd495caed7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0f68d86da14ce39e0815b885b41e73" is_pre_election: true
I20260926 07:32:05.374356 20602 raft_consensus.cc:2454] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196f56f7c2834f44a14afd495caed7e1 in term 0.
I20260926 07:32:05.374536 20732 raft_consensus.cc:2454] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196f56f7c2834f44a14afd495caed7e1 in term 0.
I20260926 07:32:05.375089 20404 leader_election.cc:302] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196f56f7c2834f44a14afd495caed7e1, 78f0cca46945493b8fc962ac8d544638; no voters:
I20260926 07:32:05.375511 20801 raft_consensus.cc:2781] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:05.375633 20801 raft_consensus.cc:482] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:05.375702 20801 raft_consensus.cc:3037] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:05.377779 20801 raft_consensus.cc:504] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.378437 20801 leader_election.cc:288] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9c0f68d86da14ce39e0815b885b41e73 (127.2.147.195:37035), 78f0cca46945493b8fc962ac8d544638 (127.2.147.194:33925)
I20260926 07:32:05.378768 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d470e214ad4346a28f07d8dcc51606" candidate_uuid: "196f56f7c2834f44a14afd495caed7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0f68d86da14ce39e0815b885b41e73"
I20260926 07:32:05.378906 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d470e214ad4346a28f07d8dcc51606" candidate_uuid: "196f56f7c2834f44a14afd495caed7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f0cca46945493b8fc962ac8d544638"
I20260926 07:32:05.378988 20732 raft_consensus.cc:3037] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:05.379134 20602 raft_consensus.cc:3037] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:05.381197 20732 raft_consensus.cc:2454] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196f56f7c2834f44a14afd495caed7e1 in term 1.
I20260926 07:32:05.381390 20602 raft_consensus.cc:2454] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196f56f7c2834f44a14afd495caed7e1 in term 1.
I20260926 07:32:05.381599 20405 leader_election.cc:302] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196f56f7c2834f44a14afd495caed7e1, 9c0f68d86da14ce39e0815b885b41e73; no voters:
I20260926 07:32:05.381888 20801 raft_consensus.cc:2781] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:05.382344 20801 raft_consensus.cc:686] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 196f56f7c2834f44a14afd495caed7e1, State: Running, Role: LEADER
I20260926 07:32:05.383049 20801 consensus_queue.cc:237] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } }
I20260926 07:32:05.387696 20329 catalog_manager.cc:5495] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 196f56f7c2834f44a14afd495caed7e1 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "196f56f7c2834f44a14afd495caed7e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196f56f7c2834f44a14afd495caed7e1" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 45889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 } health_report { overall_health: UNKNOWN } } }
I20260926 07:32:05.428640 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.423193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.2.147.193:45889)
0926 07:32:05.423428 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.423436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.423560 (+ 124us) server_negotiation.cc:407] Connection header received
0926 07:32:05.423744 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.423748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.423830 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.423969 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:05.424277 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.424966 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.426470 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.426874 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.426971 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.427561 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:05.428079 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:05.428196 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.428275 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.428335 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:32:05.437075 20732 raft_consensus.cc:1261] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 1 FOLLOWER]: Refusing update from remote peer 196f56f7c2834f44a14afd495caed7e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:05.437075 20602 raft_consensus.cc:1261] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 1 FOLLOWER]: Refusing update from remote peer 196f56f7c2834f44a14afd495caed7e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:05.438027 20806 consensus_queue.cc:1035] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0f68d86da14ce39e0815b885b41e73" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:05.438376 20801 consensus_queue.cc:1035] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f0cca46945493b8fc962ac8d544638" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:05.449739 20809 mvcc.cc:205] Tried to move back new op lower bound from 7333510862579396608 to 7333510862372651008. Current Snapshot: MvccSnapshot[applied={T|T < 7333510862579396608}]
I20260926 07:32:05.451275 20811 mvcc.cc:205] Tried to move back new op lower bound from 7333510862579396608 to 7333510862372651008. Current Snapshot: MvccSnapshot[applied={T|T < 7333510862579396608}]
I20260926 07:32:05.463671 20330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:05.463925 20330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:05.467384 20330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14b2226b96a41e99ca237e1619e2852: Sending DeleteTablet for 3 replicas of tablet e3d470e214ad4346a28f07d8dcc51606
I20260926 07:32:05.468482 20712 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d470e214ad4346a28f07d8dcc51606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:05 UTC) from {username='slave'} at 127.0.0.1:47150
I20260926 07:32:05.468482 20451 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d470e214ad4346a28f07d8dcc51606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:05 UTC) from {username='slave'} at 127.0.0.1:43302
I20260926 07:32:05.469583 20582 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d470e214ad4346a28f07d8dcc51606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:05 UTC) from {username='slave'} at 127.0.0.1:39252
I20260926 07:32:05.470997 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20389
I20260926 07:32:05.479508 20821 tablet_replica.cc:330] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: stopping tablet replica
I20260926 07:32:05.480823 20821 raft_consensus.cc:2229] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:05.481172 20820 tablet_replica.cc:330] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: stopping tablet replica
I20260926 07:32:05.481812 20819 tablet_replica.cc:330] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: stopping tablet replica
I20260926 07:32:05.482050 20821 raft_consensus.cc:2258] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:05.482398 20820 raft_consensus.cc:2229] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:05.482872 20820 raft_consensus.cc:2258] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:05.483497 20819 raft_consensus.cc:2229] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:05.484131 20819 raft_consensus.cc:2258] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:05.484750 20821 ts_tablet_manager.cc:1905] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:05.486037 20820 ts_tablet_manager.cc:1905] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:05.487303 20819 ts_tablet_manager.cc:1905] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:05.491636 20821 ts_tablet_manager.cc:1918] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:05.491879 20821 log.cc:1196] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/e3d470e214ad4346a28f07d8dcc51606
I20260926 07:32:05.492609 20821 ts_tablet_manager.cc:1939] T e3d470e214ad4346a28f07d8dcc51606 P 78f0cca46945493b8fc962ac8d544638: Deleting consensus metadata
I20260926 07:32:05.494603 20314 catalog_manager.cc:4841] TS 78f0cca46945493b8fc962ac8d544638 (127.2.147.194:33925): tablet e3d470e214ad4346a28f07d8dcc51606 (table test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]) successfully deleted
I20260926 07:32:05.496718 20820 ts_tablet_manager.cc:1918] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:05.496886 20820 log.cc:1196] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/e3d470e214ad4346a28f07d8dcc51606
I20260926 07:32:05.497428 20820 ts_tablet_manager.cc:1939] T e3d470e214ad4346a28f07d8dcc51606 P 9c0f68d86da14ce39e0815b885b41e73: Deleting consensus metadata
I20260926 07:32:05.499344 20315 catalog_manager.cc:4841] TS 9c0f68d86da14ce39e0815b885b41e73 (127.2.147.195:37035): tablet e3d470e214ad4346a28f07d8dcc51606 (table test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]) successfully deleted
I20260926 07:32:05.499904 20819 ts_tablet_manager.cc:1918] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:05.500084 20819 log.cc:1196] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/e3d470e214ad4346a28f07d8dcc51606
I20260926 07:32:05.500635 20819 ts_tablet_manager.cc:1939] T e3d470e214ad4346a28f07d8dcc51606 P 196f56f7c2834f44a14afd495caed7e1: Deleting consensus metadata
I20260926 07:32:05.502378 20317 catalog_manager.cc:4841] TS 196f56f7c2834f44a14afd495caed7e1 (127.2.147.193:45889): tablet e3d470e214ad4346a28f07d8dcc51606 (table test-table [id=9bbc44ae24cd46fea4f0a9af2d5be57c]) successfully deleted
I20260926 07:32:05.502516 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.473918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.2.147.193:45889)
0926 07:32:05.474091 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.474097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.487718 (+ 13621us) server_negotiation.cc:407] Connection header received
0926 07:32:05.488200 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.488206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.488309 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.488407 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:05.489143 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.490398 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.492674 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.493382 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.493492 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.500757 (+ 7265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:05.500768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:05.500790 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:05.501646 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:05.501723 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:05.501752 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:05.501871 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.502070 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.502183 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:32:05.503793 20511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:05.601337 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20389
I20260926 07:32:05.617491 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20520
I20260926 07:32:05.624075 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.618250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.2.147.194:33925)
0926 07:32:05.618425 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.618429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.618507 (+ 78us) server_negotiation.cc:407] Connection header received
0926 07:32:05.618675 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.618679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.618773 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.618869 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:05.619324 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.620023 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.621260 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.621638 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.621730 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.622694 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:05.622703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:05.622718 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:05.623351 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:05.623399 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:05.623419 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:05.623517 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.623628 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.623714 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:05.624722 20642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:05.713689 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20520
I20260926 07:32:05.730957 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20650
I20260926 07:32:05.738008 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:05.731824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.2.147.195:37035)
0926 07:32:05.732028 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:05.732035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:05.732055 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:32:05.732261 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:05.732267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:05.732379 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:05.732498 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:05.732924 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.733663 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.734939 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:05.735452 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:05.735596 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:05.736667 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:05.736675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:05.736691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:05.737314 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:05.737362 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:05.737383 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:05.737475 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:05.737585 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:32:05.737671 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:32:05.738687 20772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:05.827862 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20650
I20260926 07:32:05.845165 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20299
I20260926 07:32:05.846240 20360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:05.956257 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20299
2026-09-26T07:32:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2780 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:06 dist-test-slave-1dwd krb5kdc[20829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:06 dist-test-slave-1dwd krb5kdc[20829](info): set up 2 sockets
Sep 26 07:32:06 dist-test-slave-1dwd krb5kdc[20829](info): commencing operation
krb5kdc: starting...
W20260926 07:32:07.071334 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:07 dist-test-slave-1dwd krb5kdc[20829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:07.326596 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34263
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:34263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:07.663410 20842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:07.663740 20842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:07.663929 20842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:07.671234 20842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:07.671347 20842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:07.671460 20842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:07.671562 20842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:07.671648 20842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:07.681550 20842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:34263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:34263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:07.682943 20842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:07.684373 20842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:07.696208 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:07.696364 20847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:07.696393 20850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:07.697247 20842 server_base.cc:1034] running on GCE node
I20260926 07:32:07.698040 20842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:07.699481 20842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:07.700717 20842 hybrid_clock.cc:648] HybridClock initialized: now 1790407927700677 us; error 47 us; skew 500 ppm
I20260926 07:32:07.701107 20842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:07 dist-test-slave-1dwd krb5kdc[20829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:07.704305 20842 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:07.706168 20842 webserver.cc:460] Webserver started at http://127.2.147.254:38751/ using document root <none> and password file <none>
I20260926 07:32:07.706895 20842 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:07.707050 20842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:07.707440 20842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:07.710517 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "13727eb2be34433ab86487f09f4b7f36"
format_stamp: "Formatted at 2026-09-26 07:32:07 on dist-test-slave-1dwd"
server_key: "f76bac79d44fff1f6fdb3f10c76a7840"
server_key_iv: "b6939325d16d5ca13ba9e471604ff5f3"
server_key_version: "encryptionkey@0"
I20260926 07:32:07.711395 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "13727eb2be34433ab86487f09f4b7f36"
format_stamp: "Formatted at 2026-09-26 07:32:07 on dist-test-slave-1dwd"
server_key: "f76bac79d44fff1f6fdb3f10c76a7840"
server_key_iv: "b6939325d16d5ca13ba9e471604ff5f3"
server_key_version: "encryptionkey@0"
I20260926 07:32:07.716631 20842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 07:32:07.720187 20857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:07.721783 20842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:07.722024 20842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "13727eb2be34433ab86487f09f4b7f36"
format_stamp: "Formatted at 2026-09-26 07:32:07 on dist-test-slave-1dwd"
server_key: "f76bac79d44fff1f6fdb3f10c76a7840"
server_key_iv: "b6939325d16d5ca13ba9e471604ff5f3"
server_key_version: "encryptionkey@0"
I20260926 07:32:07.722254 20842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:07.739163 20842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:07.743908 20842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:07.744220 20842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:07.762938 20842 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:34263
I20260926 07:32:07.762972 20908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:34263 every 8 connection(s)
I20260926 07:32:07.764436 20842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:07.768328 20909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:07.774773 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20842
I20260926 07:32:07.774984 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:07.775445 2639 external_mini_cluster.cc:1428] Setting key dd418653fe65d53545f1153aed40526a
I20260926 07:32:07.779753 20909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: Bootstrap starting.
I20260926 07:32:07.782867 20909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:07.783983 20909 log.cc:824] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:07.786976 20909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: No bootstrap required, opened a new log
Sep 26 07:32:07 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:07.795258 20909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } }
I20260926 07:32:07.795698 20909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:07.795817 20909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13727eb2be34433ab86487f09f4b7f36, State: Initialized, Role: FOLLOWER
I20260926 07:32:07.796433 20909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } }
I20260926 07:32:07.796705 20909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:07.796862 20909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:07.797068 20909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:07.799047 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:07.777319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.2.147.254:34263)
0926 07:32:07.778141 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:07.778153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:07.778196 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:32:07.780032 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:07.780055 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:07.780439 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:07.780833 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:07.782334 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:07.783796 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:07.785622 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:07.786312 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:07.786524 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:07.790930 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:07.790938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:07.790954 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:07.790992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:07.794428 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:07.795169 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:07.795176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:07.795183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:07.795410 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:07.795897 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:07.795902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:07.795906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:07.796569 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:07.796886 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:07.797541 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 07:32:07.797961 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":355,"threads_started":1}
I20260926 07:32:07.800136 20909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } }
I20260926 07:32:07.800709 20909 leader_election.cc:302] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13727eb2be34433ab86487f09f4b7f36; no voters:
I20260926 07:32:07.801311 20909 leader_election.cc:288] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:07.801457 20914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:07.803367 20914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [term 1 LEADER]: Becoming Leader. State: Replica: 13727eb2be34433ab86487f09f4b7f36, State: Running, Role: LEADER
I20260926 07:32:07.803443 20909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:07.803869 20914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } }
I20260926 07:32:07.806617 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13727eb2be34433ab86487f09f4b7f36. Latest consensus state: current_term: 1 leader_uuid: "13727eb2be34433ab86487f09f4b7f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } } }
I20260926 07:32:07.806715 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13727eb2be34433ab86487f09f4b7f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13727eb2be34433ab86487f09f4b7f36" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 34263 } } }
I20260926 07:32:07.807078 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:07.807106 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:07.811185 20923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:07.818958 20923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:07.828791 20923 catalog_manager.cc:1347] Generated new cluster ID: 27a5a5ba41484f98ac7f10076b09550a
I20260926 07:32:07.828917 20923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:07.865494 20923 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:07.866945 20923 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:07.878360 20923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: Generated new TSK 0
I20260926 07:32:07.879349 20923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:32:07.939458 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:08.285154 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:08.285449 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:08.285624 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:08.293226 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:08.293342 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:08.293545 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:08.303695 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:08.305164 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:08.306486 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:08.320437 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:08.320508 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:08.320531 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:08.321380 20935 server_base.cc:1034] running on GCE node
I20260926 07:32:08.322044 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:08.322944 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:08.324144 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790407928324072 us; error 61 us; skew 500 ppm
I20260926 07:32:08.324425 20935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:08 dist-test-slave-1dwd krb5kdc[20829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:08.327656 20935 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:32:08.329121 20935 webserver.cc:460] Webserver started at http://127.2.147.193:41527/ using document root <none> and password file <none>
I20260926 07:32:08.329826 20935 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:08.329969 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:08.330288 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:08.333284 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "22a2058a0bcd46688e93cceb3c8d9154"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "316eac8c0cdb07567ff69a13fe563a13"
server_key_iv: "ce0164bd21f6f3dae1bb06abce880041"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.334067 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "22a2058a0bcd46688e93cceb3c8d9154"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "316eac8c0cdb07567ff69a13fe563a13"
server_key_iv: "ce0164bd21f6f3dae1bb06abce880041"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.339125 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:32:08.343370 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.344765 20935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:32:08.344941 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "22a2058a0bcd46688e93cceb3c8d9154"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "316eac8c0cdb07567ff69a13fe563a13"
server_key_iv: "ce0164bd21f6f3dae1bb06abce880041"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.345168 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:08.368995 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:08.373011 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:08.373325 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:08.374297 20935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:08.375716 20935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:08.375816 20935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.375955 20935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:08.376024 20935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.409626 20935 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33717
I20260926 07:32:08.409670 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33717 every 8 connection(s)
I20260926 07:32:08.411062 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:08.416996 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 20935
I20260926 07:32:08.417179 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:08.417749 2639 external_mini_cluster.cc:1428] Setting key 1b4486a626f12d7c55dcb039d47c1039
Sep 26 07:32:08 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:08.435156 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:08.413757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:39597 (local address 127.2.147.254:34263)
0926 07:32:08.414193 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:08.414199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:08.415284 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 07:32:08.417384 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:08.417390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:08.417458 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:08.417563 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:08.419543 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.420324 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.423020 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.423489 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.423594 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:08.428090 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:08.428095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:08.428098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:08.428134 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:08.431657 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:08.432485 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:08.432494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:08.432501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:08.432659 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:08.433083 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:08.433090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:08.433096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:08.433484 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:08.433646 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:08.434385 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 07:32:08.434661 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":195,"threads_started":1}
I20260926 07:32:08.435879 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:08.414423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34263 (local address 127.2.147.193:39597)
0926 07:32:08.415107 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:08.415147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:08.417084 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:08.417841 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:08.417855 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:08.418568 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:08.419310 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:08.419331 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.420532 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.420540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:08.422846 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:08.422869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.423729 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.423739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:08.424085 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:08.424865 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:08.424899 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:08.427828 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:08.431834 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:08.431842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:08.431858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:08.432331 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:08.432769 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:08.432771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:08.432774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:08.432957 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:08.433616 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:08.433621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:08.433969 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 07:32:08.434218 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 07:32:08.434638 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
I20260926 07:32:08.437568 21063 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34263
I20260926 07:32:08.437924 21063 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:08.438815 21063 heartbeater.cc:511] Master 127.2.147.254:34263 requested a full tablet report, sending...
I20260926 07:32:08.441263 20874 ts_manager.cc:194] Registered new tserver with Master: 22a2058a0bcd46688e93cceb3c8d9154 (127.2.147.193:33717)
I20260926 07:32:08.442749 20874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:39597
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:32:08.478025 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:08.805562 21069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:08.805923 21069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:08.806113 21069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:08.813319 21069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:08.813426 21069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:08.813635 21069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:08.823421 21069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:08.824765 21069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:08.826184 21069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:08.839474 21077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:08.839532 21075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:08.839439 21074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:08.840718 21069 server_base.cc:1034] running on GCE node
I20260926 07:32:08.841276 21069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:08.842154 21069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:08.843403 21069 hybrid_clock.cc:648] HybridClock initialized: now 1790407928843349 us; error 47 us; skew 500 ppm
I20260926 07:32:08.843817 21069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:08 dist-test-slave-1dwd krb5kdc[20829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:08.847293 21069 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:32:08.848809 21069 webserver.cc:460] Webserver started at http://127.2.147.194:44471/ using document root <none> and password file <none>
I20260926 07:32:08.849550 21069 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:08.849680 21069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:08.849996 21069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:08.853015 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "75eee6e0538445bd94cfe493aab04ce9"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "017b7fc6b427ad877b09f3d35165f46e"
server_key_iv: "93f001f0dbc5c12096528cc97302db20"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.853793 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "75eee6e0538445bd94cfe493aab04ce9"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "017b7fc6b427ad877b09f3d35165f46e"
server_key_iv: "93f001f0dbc5c12096528cc97302db20"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.859310 21069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:32:08.862756 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.864189 21069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:32:08.864327 21069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "75eee6e0538445bd94cfe493aab04ce9"
format_stamp: "Formatted at 2026-09-26 07:32:08 on dist-test-slave-1dwd"
server_key: "017b7fc6b427ad877b09f3d35165f46e"
server_key_iv: "93f001f0dbc5c12096528cc97302db20"
server_key_version: "encryptionkey@0"
I20260926 07:32:08.864495 21069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:08.874828 21069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:08.878829 21069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:08.879097 21069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:08.880060 21069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:08.881359 21069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:08.881433 21069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.881513 21069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:08.881558 21069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:08.915087 21069 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:38663
I20260926 07:32:08.915105 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:38663 every 8 connection(s)
I20260926 07:32:08.916409 21069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:08.922030 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21069
I20260926 07:32:08.922215 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:08.922720 2639 external_mini_cluster.cc:1428] Setting key 2b5155ec9e0d87ad5123d9f97b4fde44
Sep 26 07:32:08 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:08.938144 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:08.918878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55571 (local address 127.2.147.254:34263)
0926 07:32:08.919062 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:08.919066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:08.920268 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 07:32:08.922392 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:08.922397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:08.922472 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:08.922562 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:08.924400 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.925181 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.927743 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.928176 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.928273 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:08.932434 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:08.932439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:08.932442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:08.932473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:08.934991 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:08.935792 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:08.935798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:08.935800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:08.935942 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:08.936414 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:08.936417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:08.936419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:08.936684 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:08.936842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:08.937545 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 07:32:08.937784 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:32:08.939111 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:08.919433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34263 (local address 127.2.147.194:55571)
0926 07:32:08.920113 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:08.920150 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:08.922041 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:08.922831 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:08.922845 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:08.923526 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:08.924233 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:08.924257 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.925337 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.925343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:08.927597 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:08.927620 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:08.928407 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:08.928416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:08.928741 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:08.929476 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:08.929509 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:08.932190 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:08.935142 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:08.935151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:08.935168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:08.935652 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:08.936058 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:08.936064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:08.936068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:08.936267 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:08.936832 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:08.936840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:08.937181 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 07:32:08.937455 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 07:32:08.937988 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":260,"threads_started":1}
I20260926 07:32:08.940646 21197 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34263
I20260926 07:32:08.940960 21197 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:08.941670 21197 heartbeater.cc:511] Master 127.2.147.254:34263 requested a full tablet report, sending...
I20260926 07:32:08.943161 20874 ts_manager.cc:194] Registered new tserver with Master: 75eee6e0538445bd94cfe493aab04ce9 (127.2.147.194:38663)
I20260926 07:32:08.944454 20874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:55571
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:32:08.982604 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:09.306020 21202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:09.306298 21202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:09.306490 21202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:09.313755 21202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:09.313855 21202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:09.314002 21202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:09.323537 21202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:34263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:09.324836 21202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:09.326277 21202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:09.339619 21210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:09.339622 21207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:09.339572 21208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:09.340306 21202 server_base.cc:1034] running on GCE node
I20260926 07:32:09.340912 21202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:09.341761 21202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:09.342957 21202 hybrid_clock.cc:648] HybridClock initialized: now 1790407929342895 us; error 51 us; skew 500 ppm
I20260926 07:32:09.343333 21202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:09 dist-test-slave-1dwd krb5kdc[20829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407929, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:09.347936 21202 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:32:09.349541 21202 webserver.cc:460] Webserver started at http://127.2.147.195:45775/ using document root <none> and password file <none>
I20260926 07:32:09.350317 21202 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:09.350461 21202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:09.350790 21202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:09.353767 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "1a80b594238645a2a39ab769f1a65136"
format_stamp: "Formatted at 2026-09-26 07:32:09 on dist-test-slave-1dwd"
server_key: "6933819f62bbd77577c6fd65c9d08d97"
server_key_iv: "700bcb0f9ef00e9e7a2cda2803fbcc51"
server_key_version: "encryptionkey@0"
I20260926 07:32:09.354566 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "1a80b594238645a2a39ab769f1a65136"
format_stamp: "Formatted at 2026-09-26 07:32:09 on dist-test-slave-1dwd"
server_key: "6933819f62bbd77577c6fd65c9d08d97"
server_key_iv: "700bcb0f9ef00e9e7a2cda2803fbcc51"
server_key_version: "encryptionkey@0"
I20260926 07:32:09.359757 21202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 07:32:09.363322 21217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:09.364667 21202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:09.364848 21202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "1a80b594238645a2a39ab769f1a65136"
format_stamp: "Formatted at 2026-09-26 07:32:09 on dist-test-slave-1dwd"
server_key: "6933819f62bbd77577c6fd65c9d08d97"
server_key_iv: "700bcb0f9ef00e9e7a2cda2803fbcc51"
server_key_version: "encryptionkey@0"
I20260926 07:32:09.365075 21202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:09.380883 21202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:09.385221 21202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:09.385524 21202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:09.386576 21202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:09.387993 21202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:09.388108 21202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:09.388264 21202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:09.388336 21202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:09.421695 21202 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:34437
I20260926 07:32:09.421734 21329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:34437 every 8 connection(s)
I20260926 07:32:09.423120 21202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:09.427489 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21202
I20260926 07:32:09.427676 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:09.428123 2639 external_mini_cluster.cc:1428] Setting key 4319abb54891fd5f5decd74fe3faa7bd
Sep 26 07:32:09 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407929, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:09.445066 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.425796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:42461 (local address 127.2.147.254:34263)
0926 07:32:09.425976 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.425981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.427336 (+ 1355us) server_negotiation.cc:407] Connection header received
0926 07:32:09.429602 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.429607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.429700 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.429807 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:09.431772 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.432526 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.433973 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.434362 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.434451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.438955 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:09.438959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:09.438962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:09.438994 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:09.441888 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:09.442731 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:09.442737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:09.442747 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:09.442882 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:09.443355 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:09.443359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:09.443361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:09.443619 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:09.443780 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.444483 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.444708 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:09.445990 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.426384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:34263 (local address 127.2.147.195:42461)
0926 07:32:09.427119 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.427157 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.429328 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.430107 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.430120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.430924 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:09.431593 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.431611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.432651 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.432655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.433853 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.433865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.434541 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.434545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.434770 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.436498 (+ 1728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:09.436520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:09.438744 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:09.442026 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:09.442034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:09.442050 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:09.442605 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:09.442995 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:09.443008 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:09.443011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:09.443246 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:09.443751 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:09.443758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:09.444117 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.444390 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.444893 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260926 07:32:09.446723 21063 heartbeater.cc:503] Master 127.2.147.254:34263 was elected leader, sending a full tablet report...
I20260926 07:32:09.447564 21330 heartbeater.cc:347] Connected to a master server at 127.2.147.254:34263
I20260926 07:32:09.447901 21330 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:09.448693 21330 heartbeater.cc:511] Master 127.2.147.254:34263 requested a full tablet report, sending...
I20260926 07:32:09.449958 20874 ts_manager.cc:194] Registered new tserver with Master: 1a80b594238645a2a39ab769f1a65136 (127.2.147.195:34437)
I20260926 07:32:09.450738 20874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:42461
I20260926 07:32:09.455343 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:09.472762 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.459162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.2.147.254:34263)
0926 07:32:09.459381 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.459388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.459783 (+ 395us) server_negotiation.cc:407] Connection header received
0926 07:32:09.459970 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.459976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.460042 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.460143 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:09.461507 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.462226 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.463721 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.464141 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.464275 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.466942 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:09.466956 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:09.466961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:09.466993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:09.469802 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:09.470619 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:09.470625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:09.470629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:09.470758 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:09.471309 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:09.471313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:09.471315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:09.471606 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:09.471901 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.472279 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.472469 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 07:32:09.476676 20874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:09.483484 20874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:09.509277 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.496034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:38663 (local address 127.0.0.1:44342)
0926 07:32:09.496790 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.496828 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.496998 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.498269 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.498277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.498306 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:09.498751 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.498764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.501258 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.501267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.504291 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.504313 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.507598 (+ 3285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.507607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.507860 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.507923 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.508060 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.508295 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":705,"threads_started":1}
I20260926 07:32:09.509544 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.496594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.2.147.194:38663)
0926 07:32:09.497610 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.497620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.497648 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:32:09.497761 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.497770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.498034 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.498953 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:09.499183 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.501084 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.504616 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.506546 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.508536 (+ 1990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.508758 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.508963 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.509166 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":2245,"threads_started":1}
I20260926 07:32:09.513602 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9ffc226374e92a17ec179acdf347d (DEFAULT_TABLE table=test-table [id=af24dd5ff0734e35998c67f69479d6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:09.515163 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd9ffc226374e92a17ec179acdf347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:09.516240 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.496110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:34437 (local address 127.0.0.1:55818)
0926 07:32:09.507269 (+ 11159us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.507290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.507423 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.508244 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.508252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.508291 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:09.509235 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.509252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.511088 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.511092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.513298 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.513311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.515409 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.515414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.515589 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.515620 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.515714 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.515863 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11012,"thread_start_us":10703,"threads_started":1}
I20260926 07:32:09.517114 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.496144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.2.147.195:34437)
0926 07:32:09.507504 (+ 11360us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.507517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.507558 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:09.507678 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.507689 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.507976 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.508550 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:09.509481 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.510938 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.513489 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.514884 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.515013 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.515270 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.516559 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.516736 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11107,"thread_start_us":10925,"threads_started":1}
I20260926 07:32:09.517799 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.497727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33717 (local address 127.0.0.1:47138)
0926 07:32:09.509057 (+ 11330us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.509076 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.509200 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.510333 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.510339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.510407 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:09.511372 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.511387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.513752 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.513758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.515208 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.515241 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.517068 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.517074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.517239 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.517270 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.517353 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.517493 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10739,"thread_start_us":10561,"threads_started":1}
I20260926 07:32:09.517938 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.498620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.2.147.193:33717)
0926 07:32:09.508124 (+ 9504us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.508132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.509332 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 07:32:09.509862 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.509870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.510114 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.510323 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:09.511572 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.513618 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.515414 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.516850 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.516995 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.517222 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.517444 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.517610 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9318,"thread_start_us":9141,"threads_started":1}
I20260926 07:32:09.521466 20998 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9ffc226374e92a17ec179acdf347d (DEFAULT_TABLE table=test-table [id=af24dd5ff0734e35998c67f69479d6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:09.521585 21265 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd9ffc226374e92a17ec179acdf347d (DEFAULT_TABLE table=test-table [id=af24dd5ff0734e35998c67f69479d6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:09.522786 20998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd9ffc226374e92a17ec179acdf347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:09.523010 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd9ffc226374e92a17ec179acdf347d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:09.531769 21348 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Bootstrap starting.
I20260926 07:32:09.535109 21349 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Bootstrap starting.
I20260926 07:32:09.535494 21348 tablet_bootstrap.cc:654] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:09.536787 21348 log.cc:824] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:09.537537 21350 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Bootstrap starting.
I20260926 07:32:09.539009 21349 tablet_bootstrap.cc:654] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:09.540759 21349 log.cc:824] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:09.540934 21350 tablet_bootstrap.cc:654] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:09.542160 21350 log.cc:824] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:09.544051 21348 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: No bootstrap required, opened a new log
I20260926 07:32:09.544047 21349 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: No bootstrap required, opened a new log
I20260926 07:32:09.544325 21349 ts_tablet_manager.cc:1397] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:32:09.544337 21348 ts_tablet_manager.cc:1397] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260926 07:32:09.544713 21350 tablet_bootstrap.cc:492] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: No bootstrap required, opened a new log
I20260926 07:32:09.544953 21350 ts_tablet_manager.cc:1397] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:32:09.552637 21350 raft_consensus.cc:348] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.553042 21350 raft_consensus.cc:374] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:09.553138 21350 raft_consensus.cc:729] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a80b594238645a2a39ab769f1a65136, State: Initialized, Role: FOLLOWER
I20260926 07:32:09.553776 21350 consensus_queue.cc:260] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.554877 21330 heartbeater.cc:503] Master 127.2.147.254:34263 was elected leader, sending a full tablet report...
I20260926 07:32:09.555598 21350 ts_tablet_manager.cc:1428] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 07:32:09.555327 21349 raft_consensus.cc:348] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.555325 21348 raft_consensus.cc:348] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.555756 21348 raft_consensus.cc:374] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:09.555760 21349 raft_consensus.cc:374] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:09.555850 21349 raft_consensus.cc:729] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a2058a0bcd46688e93cceb3c8d9154, State: Initialized, Role: FOLLOWER
I20260926 07:32:09.555910 21348 raft_consensus.cc:729] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eee6e0538445bd94cfe493aab04ce9, State: Initialized, Role: FOLLOWER
I20260926 07:32:09.556535 21348 consensus_queue.cc:260] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.556525 21349 consensus_queue.cc:260] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.558110 21197 heartbeater.cc:503] Master 127.2.147.254:34263 was elected leader, sending a full tablet report...
I20260926 07:32:09.558662 21348 ts_tablet_manager.cc:1428] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 07:32:09.560217 21349 ts_tablet_manager.cc:1428] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260926 07:32:09.663846 21064 tablet.cc:2369] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:32:09.668367 21198 tablet.cc:2369] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:32:09.674786 21331 tablet.cc:2369] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:09.869576 21355 raft_consensus.cc:482] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:09.869854 21355 raft_consensus.cc:504] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.871635 21355 leader_election.cc:288] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a80b594238645a2a39ab769f1a65136 (127.2.147.195:34437), 22a2058a0bcd46688e93cceb3c8d9154 (127.2.147.193:33717)
I20260926 07:32:09.880896 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.872200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:58193 (local address 127.2.147.195:34437)
0926 07:32:09.872371 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.872379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.873061 (+ 682us) server_negotiation.cc:407] Connection header received
0926 07:32:09.873273 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.873278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.873360 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.873462 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:09.874278 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.875420 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.878415 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.879729 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.879877 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.879953 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.880513 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.880641 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:32:09.880900 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.872107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:55497 (local address 127.2.147.193:33717)
0926 07:32:09.872324 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.872331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.873061 (+ 730us) server_negotiation.cc:407] Connection header received
0926 07:32:09.873273 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.873279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.873360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.873462 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:09.874278 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.875422 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.878415 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.879698 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.879877 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.879953 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.880508 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.880643 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:32:09.881150 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.871956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:34437 (local address 127.2.147.194:58193)
0926 07:32:09.872913 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.872938 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.873120 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.873493 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.873498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.873531 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:09.874091 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.874109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.875553 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.875559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.878245 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.878258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.879998 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.880003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.880301 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.880366 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.880465 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.880661 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":264,"spinlock_wait_cycles":15232,"thread_start_us":573,"threads_started":1}
I20260926 07:32:09.881474 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.872044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33717 (local address 127.2.147.194:55497)
0926 07:32:09.872940 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:09.872955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:09.873120 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:09.873493 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:09.873498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:09.873532 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:09.874131 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.874140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.875555 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.875559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.878243 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:09.878258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.879995 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.880002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:09.880296 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.880367 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:32:09.880465 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:32:09.880661 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":309,"spinlock_wait_cycles":31488,"thread_start_us":483,"threads_started":1}
I20260926 07:32:09.882153 21285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9ffc226374e92a17ec179acdf347d" candidate_uuid: "75eee6e0538445bd94cfe493aab04ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a80b594238645a2a39ab769f1a65136" is_pre_election: true
I20260926 07:32:09.882368 21018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9ffc226374e92a17ec179acdf347d" candidate_uuid: "75eee6e0538445bd94cfe493aab04ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a2058a0bcd46688e93cceb3c8d9154" is_pre_election: true
I20260926 07:32:09.882653 21285 raft_consensus.cc:2454] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eee6e0538445bd94cfe493aab04ce9 in term 0.
I20260926 07:32:09.882915 21018 raft_consensus.cc:2454] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eee6e0538445bd94cfe493aab04ce9 in term 0.
I20260926 07:32:09.883309 21086 leader_election.cc:302] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a80b594238645a2a39ab769f1a65136, 75eee6e0538445bd94cfe493aab04ce9; no voters:
I20260926 07:32:09.883780 21355 raft_consensus.cc:2781] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:09.883934 21355 raft_consensus.cc:482] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:09.884049 21355 raft_consensus.cc:3037] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:09.886153 21355 raft_consensus.cc:504] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.886771 21355 leader_election.cc:288] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a80b594238645a2a39ab769f1a65136 (127.2.147.195:34437), 22a2058a0bcd46688e93cceb3c8d9154 (127.2.147.193:33717)
I20260926 07:32:09.887144 21285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9ffc226374e92a17ec179acdf347d" candidate_uuid: "75eee6e0538445bd94cfe493aab04ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a80b594238645a2a39ab769f1a65136"
I20260926 07:32:09.887404 21285 raft_consensus.cc:3037] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:09.887593 21018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd9ffc226374e92a17ec179acdf347d" candidate_uuid: "75eee6e0538445bd94cfe493aab04ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a2058a0bcd46688e93cceb3c8d9154"
I20260926 07:32:09.887809 21018 raft_consensus.cc:3037] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:09.890633 21285 raft_consensus.cc:2454] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eee6e0538445bd94cfe493aab04ce9 in term 1.
I20260926 07:32:09.890924 21018 raft_consensus.cc:2454] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eee6e0538445bd94cfe493aab04ce9 in term 1.
I20260926 07:32:09.891069 21086 leader_election.cc:302] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a80b594238645a2a39ab769f1a65136, 75eee6e0538445bd94cfe493aab04ce9; no voters:
I20260926 07:32:09.891398 21355 raft_consensus.cc:2781] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:09.891855 21355 raft_consensus.cc:686] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 75eee6e0538445bd94cfe493aab04ce9, State: Running, Role: LEADER
I20260926 07:32:09.892356 21355 consensus_queue.cc:237] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } }
I20260926 07:32:09.897140 20872 catalog_manager.cc:5495] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75eee6e0538445bd94cfe493aab04ce9 (127.2.147.194). New cstate: current_term: 1 leader_uuid: "75eee6e0538445bd94cfe493aab04ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eee6e0538445bd94cfe493aab04ce9" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 } health_report { overall_health: UNKNOWN } } }
I20260926 07:32:09.978261 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:09.972476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.2.147.194:38663)
0926 07:32:09.972700 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:09.972706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:09.972814 (+ 108us) server_negotiation.cc:407] Connection header received
0926 07:32:09.973085 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:09.973089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:09.973175 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:09.973325 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:09.973717 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.974410 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.975975 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:09.976379 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:09.976525 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:09.977130 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:09.977633 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:09.977822 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:09.977941 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:32:09.977990 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:09.986727 21285 raft_consensus.cc:1261] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 1 FOLLOWER]: Refusing update from remote peer 75eee6e0538445bd94cfe493aab04ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:09.986729 21018 raft_consensus.cc:1261] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 1 FOLLOWER]: Refusing update from remote peer 75eee6e0538445bd94cfe493aab04ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:09.987680 21359 consensus_queue.cc:1035] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a2058a0bcd46688e93cceb3c8d9154" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:09.988063 21355 consensus_queue.cc:1035] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a80b594238645a2a39ab769f1a65136" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:10.014272 20872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:10.014533 20872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:10.017776 20872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13727eb2be34433ab86487f09f4b7f36: Sending DeleteTablet for 3 replicas of tablet 7cd9ffc226374e92a17ec179acdf347d
I20260926 07:32:10.018857 21265 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd9ffc226374e92a17ec179acdf347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:10 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:55818
I20260926 07:32:10.019294 20998 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd9ffc226374e92a17ec179acdf347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:10 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:47138
I20260926 07:32:10.020828 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd9ffc226374e92a17ec179acdf347d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:10 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:44342
I20260926 07:32:10.021523 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20935
I20260926 07:32:10.034336 21372 tablet_replica.cc:330] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: stopping tablet replica
I20260926 07:32:10.034780 21372 raft_consensus.cc:2229] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:10.035357 21372 raft_consensus.cc:2258] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:10.039023 21372 ts_tablet_manager.cc:1905] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:10.045324 21372 ts_tablet_manager.cc:1918] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:10.045502 21372 log.cc:1196] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/7cd9ffc226374e92a17ec179acdf347d
I20260926 07:32:10.046154 21372 ts_tablet_manager.cc:1939] T 7cd9ffc226374e92a17ec179acdf347d P 22a2058a0bcd46688e93cceb3c8d9154: Deleting consensus metadata
I20260926 07:32:10.048506 20860 catalog_manager.cc:4841] TS 22a2058a0bcd46688e93cceb3c8d9154 (127.2.147.193:33717): tablet 7cd9ffc226374e92a17ec179acdf347d (table test-table [id=af24dd5ff0734e35998c67f69479d6d9]) successfully deleted
I20260926 07:32:10.049414 21373 tablet_replica.cc:330] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: stopping tablet replica
I20260926 07:32:10.050020 21373 raft_consensus.cc:2229] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:10.050808 21373 raft_consensus.cc:2258] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:10.053598 21373 ts_tablet_manager.cc:1905] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:10.055358 21371 tablet_replica.cc:330] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: stopping tablet replica
I20260926 07:32:10.055783 21371 raft_consensus.cc:2229] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:10.056319 21371 raft_consensus.cc:2258] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:10.058235 21371 ts_tablet_manager.cc:1905] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:32:10 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:32:10.062023 21373 ts_tablet_manager.cc:1918] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:10.062206 21373 log.cc:1196] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/7cd9ffc226374e92a17ec179acdf347d
I20260926 07:32:10.062778 21373 ts_tablet_manager.cc:1939] T 7cd9ffc226374e92a17ec179acdf347d P 75eee6e0538445bd94cfe493aab04ce9: Deleting consensus metadata
I20260926 07:32:10.064767 20860 catalog_manager.cc:4841] TS 75eee6e0538445bd94cfe493aab04ce9 (127.2.147.194:38663): tablet 7cd9ffc226374e92a17ec179acdf347d (table test-table [id=af24dd5ff0734e35998c67f69479d6d9]) successfully deleted
I20260926 07:32:10.065456 21371 ts_tablet_manager.cc:1918] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:10.065637 21371 log.cc:1196] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/7cd9ffc226374e92a17ec179acdf347d
I20260926 07:32:10.066171 21371 ts_tablet_manager.cc:1939] T 7cd9ffc226374e92a17ec179acdf347d P 1a80b594238645a2a39ab769f1a65136: Deleting consensus metadata
I20260926 07:32:10.068092 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:10.022838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.2.147.193:33717)
0926 07:32:10.023020 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:10.023027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:10.047327 (+ 24300us) server_negotiation.cc:407] Connection header received
0926 07:32:10.047867 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:10.047874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:10.047958 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:10.048053 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:10.050291 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.051461 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.053440 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.054072 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.054170 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:10.060742 (+ 6572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:10.060762 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:10.060780 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:10.060830 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:10.064801 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.065788 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.065795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.065797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.065941 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.066401 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.066407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.066410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.066780 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:10.067125 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:10.067437 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 07:32:10.067697 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:32:10.068058 20859 catalog_manager.cc:4841] TS 1a80b594238645a2a39ab769f1a65136 (127.2.147.195:34437): tablet 7cd9ffc226374e92a17ec179acdf347d (table test-table [id=af24dd5ff0734e35998c67f69479d6d9]) successfully deleted
I20260926 07:32:10.068980 21058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:10.157541 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20935
I20260926 07:32:10.181373 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21069
Sep 26 07:32:10 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:32:10.197192 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:10.182107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.2.147.194:38663)
0926 07:32:10.182302 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:10.182308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:10.182418 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:32:10.182615 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:10.182620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:10.182702 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:10.182816 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:10.184156 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.184957 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.186236 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.186679 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.186825 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:10.191147 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:10.191162 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:10.191181 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:10.191264 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:10.194369 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.195060 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.195077 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.195079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.195273 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.195653 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.195658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.195662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.196013 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:10.196370 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:10.196596 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:32:10.196809 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:32:10.197887 21192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:10.314429 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21069
I20260926 07:32:10.333663 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21202
Sep 26 07:32:10 dist-test-slave-1dwd krb5kdc[20829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:32:10.349043 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:10.334503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.2.147.195:34437)
0926 07:32:10.334686 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:10.334691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:10.334709 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:10.334856 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:10.334859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:10.334927 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:10.335011 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:10.336487 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.337200 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.338441 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:10.338821 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:10.338917 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:10.343294 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:10.343307 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:10.343322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:10.343359 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:10.345881 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.346532 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.346540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.346542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.346708 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:10.347108 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:10.347112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:10.347114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:10.347716 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:10.348043 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:10.348398 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 07:32:10.348620 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:32:10.349844 21325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:10.438570 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21202
I20260926 07:32:10.455171 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 20842
I20260926 07:32:10.456286 20904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:10.566088 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 20842
2026-09-26T07:32:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4610 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:10Z Disabled control of system clock
I20260926 07:32:10.642117 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38119
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:10.964833 21382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:10.965109 21382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:10.965245 21382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:10.972430 21382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:10.972534 21382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:10.972605 21382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:10.972695 21382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:10.972777 21382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:10.981885 21382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:10.983314 21382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:10.985221 21382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:10.997980 21387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:10.998035 21388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:10.998064 21390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:10.998939 21382 server_base.cc:1034] running on GCE node
I20260926 07:32:10.999817 21382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:11.001247 21382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:11.002490 21382 hybrid_clock.cc:648] HybridClock initialized: now 1790407931002446 us; error 56 us; skew 500 ppm
I20260926 07:32:11.002836 21382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:11.011613 21382 webserver.cc:460] Webserver started at http://127.2.147.254:33971/ using document root <none> and password file <none>
I20260926 07:32:11.012328 21382 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:11.012471 21382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:11.012791 21382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:11.015720 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "f5202a66b46448d18a75bb0914ffcd63"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "86d37ab9f2e85e6251417cb3da9627c9"
server_key_iv: "72af4b8d249e3da3286dcc9381c8badf"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.016538 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "f5202a66b46448d18a75bb0914ffcd63"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "86d37ab9f2e85e6251417cb3da9627c9"
server_key_iv: "72af4b8d249e3da3286dcc9381c8badf"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.021413 21382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:32:11.024894 21396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:11.026328 21382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:11.026551 21382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f5202a66b46448d18a75bb0914ffcd63"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "86d37ab9f2e85e6251417cb3da9627c9"
server_key_iv: "72af4b8d249e3da3286dcc9381c8badf"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.026777 21382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:11.055296 21382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:11.056154 21382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:11.056452 21382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:11.074692 21382 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38119
I20260926 07:32:11.074743 21447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38119 every 8 connection(s)
I20260926 07:32:11.076046 21382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:11.079697 21448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:11.085247 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21382
I20260926 07:32:11.085415 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:11.085932 2639 external_mini_cluster.cc:1428] Setting key acf95093d8c274487b6b5699f0bc0de3
I20260926 07:32:11.090764 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: Bootstrap starting.
I20260926 07:32:11.093848 21448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:11.094810 21448 log.cc:824] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:11.098047 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: No bootstrap required, opened a new log
I20260926 07:32:11.099660 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:11.087968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.2.147.254:38119)
0926 07:32:11.089043 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:11.089053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:11.089093 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:11.090708 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:11.090728 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:11.091282 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:11.091652 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:11.092242 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.093487 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.094868 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.096004 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.096189 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:11.097409 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:11.097417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:11.097430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:11.097945 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:11.098030 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:11.098053 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:11.098182 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:11.098397 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:32:11.098704 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":370,"threads_started":1}
I20260926 07:32:11.108677 21448 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } }
I20260926 07:32:11.109120 21448 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:11.109232 21448 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5202a66b46448d18a75bb0914ffcd63, State: Initialized, Role: FOLLOWER
I20260926 07:32:11.109894 21448 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } }
I20260926 07:32:11.110139 21448 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:11.110288 21448 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:11.110467 21448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:11.112984 21448 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } }
I20260926 07:32:11.113612 21448 leader_election.cc:302] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5202a66b46448d18a75bb0914ffcd63; no voters:
I20260926 07:32:11.114187 21448 leader_election.cc:288] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:11.114318 21453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:11.115036 21453 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [term 1 LEADER]: Becoming Leader. State: Replica: f5202a66b46448d18a75bb0914ffcd63, State: Running, Role: LEADER
I20260926 07:32:11.115609 21453 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } }
I20260926 07:32:11.116312 21448 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:11.118153 21454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5202a66b46448d18a75bb0914ffcd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } } }
I20260926 07:32:11.118475 21454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:11.119366 21459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:11.119170 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5202a66b46448d18a75bb0914ffcd63. Latest consensus state: current_term: 1 leader_uuid: "f5202a66b46448d18a75bb0914ffcd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5202a66b46448d18a75bb0914ffcd63" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38119 } } }
I20260926 07:32:11.119516 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:11.125072 21459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:11.134945 21459 catalog_manager.cc:1347] Generated new cluster ID: 7ac9f7bf51ac451ca49e83e9cad79632
I20260926 07:32:11.135097 21459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:11.149554 21459 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:11.150833 21459 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:11.170008 21459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: Generated new TSK 0
I20260926 07:32:11.171262 21459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:11.187702 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--builtin_ntp_servers=127.2.147.212:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:11.530373 21472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:11.530670 21472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:11.530759 21472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:11.538408 21472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:11.538518 21472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:11.538630 21472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:11.547935 21472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:11.549197 21472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:11.551065 21472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:11.564479 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:11.564513 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:11.564709 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:11.565379 21472 server_base.cc:1034] running on GCE node
I20260926 07:32:11.566015 21472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:11.567030 21472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:11.568279 21472 hybrid_clock.cc:648] HybridClock initialized: now 1790407931568233 us; error 66 us; skew 500 ppm
I20260926 07:32:11.568650 21472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:11.570983 21472 webserver.cc:460] Webserver started at http://127.2.147.193:35303/ using document root <none> and password file <none>
I20260926 07:32:11.571815 21472 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:11.571954 21472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:11.572284 21472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:11.575299 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "a184907958ab4730b830d6fd47a71b43"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "4dc64347fee502c10476c5df0a987f2a"
server_key_iv: "329c65e8cb979ebbe27e8e0a29c82362"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.576131 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "a184907958ab4730b830d6fd47a71b43"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "4dc64347fee502c10476c5df0a987f2a"
server_key_iv: "329c65e8cb979ebbe27e8e0a29c82362"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.581614 21472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:11.585230 21486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:11.586622 21472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:11.586807 21472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "a184907958ab4730b830d6fd47a71b43"
format_stamp: "Formatted at 2026-09-26 07:32:11 on dist-test-slave-1dwd"
server_key: "4dc64347fee502c10476c5df0a987f2a"
server_key_iv: "329c65e8cb979ebbe27e8e0a29c82362"
server_key_version: "encryptionkey@0"
I20260926 07:32:11.587037 21472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:11.604390 21472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:11.605285 21472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:11.605634 21472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:11.606647 21472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:11.608384 21472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:11.608531 21472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:11.608662 21472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:11.608753 21472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:11.642441 21472 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:33433
I20260926 07:32:11.642534 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:33433 every 8 connection(s)
I20260926 07:32:11.643890 21472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:11.652359 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21472
I20260926 07:32:11.652568 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:11.653018 2639 external_mini_cluster.cc:1428] Setting key 67ec696dd4cf28eb2e5ceff520b25500
I20260926 07:32:11.655879 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--builtin_ntp_servers=127.2.147.212:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:11.657361 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:11.646594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60409 (local address 127.2.147.254:38119)
0926 07:32:11.647057 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:11.647063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:11.648448 (+ 1385us) server_negotiation.cc:407] Connection header received
0926 07:32:11.650229 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:11.650233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:11.650301 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:11.650403 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:11.651487 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.652435 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.654210 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.654777 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.654871 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:11.656378 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:11.656387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:11.656394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:11.656419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:11.656474 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:11.656494 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:11.656592 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:11.656803 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:32:11.656925 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":249,"threads_started":1}
I20260926 07:32:11.658121 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:11.647283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38119 (local address 127.2.147.193:60409)
0926 07:32:11.648285 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:11.648340 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:11.649952 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:11.650616 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:11.650624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:11.650712 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:11.651340 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:11.651357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.652609 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.652631 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:11.654080 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:11.654092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:11.654996 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:11.655006 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:11.655269 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:11.656030 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:11.656035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:11.656099 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:11.656190 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:11.656595 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:11.656603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:11.656608 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:32:11.656740 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:32:11.657037 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260926 07:32:11.659699 21599 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38119
I20260926 07:32:11.660037 21599 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:11.660972 21599 heartbeater.cc:511] Master 127.2.147.254:38119 requested a full tablet report, sending...
I20260926 07:32:11.663295 21413 ts_manager.cc:194] Registered new tserver with Master: a184907958ab4730b830d6fd47a71b43 (127.2.147.193:33433)
I20260926 07:32:11.664820 21413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.193:60409
W20260926 07:32:11.986613 21603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:11.986899 21603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:11.986989 21603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:11.994424 21603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:11.994535 21603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:11.994643 21603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:12.003844 21603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:12.005100 21603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:12.006888 21603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:12.020390 21609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:12.020354 21608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:12.020435 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:12.021087 21603 server_base.cc:1034] running on GCE node
I20260926 07:32:12.021750 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:12.022753 21603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:12.024000 21603 hybrid_clock.cc:648] HybridClock initialized: now 1790407932023958 us; error 64 us; skew 500 ppm
I20260926 07:32:12.024397 21603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:12.026711 21603 webserver.cc:460] Webserver started at http://127.2.147.194:38517/ using document root <none> and password file <none>
I20260926 07:32:12.027527 21603 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:12.027662 21603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:12.028000 21603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:12.031131 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "d05b6c9b42bb43639b7ca9bc651935b1"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "1862050c2f831d2266d3571b6eef9025"
server_key_iv: "230cbb5960f6d9f18cf2830854211a8f"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.032055 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "d05b6c9b42bb43639b7ca9bc651935b1"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "1862050c2f831d2266d3571b6eef9025"
server_key_iv: "230cbb5960f6d9f18cf2830854211a8f"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.037179 21603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:12.040592 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.041878 21603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:12.042054 21603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "d05b6c9b42bb43639b7ca9bc651935b1"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "1862050c2f831d2266d3571b6eef9025"
server_key_iv: "230cbb5960f6d9f18cf2830854211a8f"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.042270 21603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:12.064980 21603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:12.065824 21603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:12.066157 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:12.067152 21603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:12.068656 21603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:12.068776 21603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.068912 21603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:12.069003 21603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.103699 21603 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:39571
I20260926 07:32:12.103785 21729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:39571 every 8 connection(s)
I20260926 07:32:12.105005 21603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:12.112731 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21603
I20260926 07:32:12.112920 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:12.113476 2639 external_mini_cluster.cc:1428] Setting key 32482f2605a937084cf97d3144c5ba0f
I20260926 07:32:12.116561 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--builtin_ntp_servers=127.2.147.212:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:12.118129 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.107667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:53561 (local address 127.2.147.254:38119)
0926 07:32:12.107871 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.107878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.109264 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 07:32:12.110957 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.110962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.111029 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.111116 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:12.112206 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.113010 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.115071 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.115646 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.115784 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.117340 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:12.117344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:12.117348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:12.117365 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:12.117396 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:12.117408 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:12.117525 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.117734 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.117832 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:32:12.119187 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.108228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38119 (local address 127.2.147.194:53561)
0926 07:32:12.109114 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.109148 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.110723 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.111369 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.111377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.111470 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:12.112055 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.112072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.113174 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.113179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.114891 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.114904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.115938 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.115945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.116167 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.117011 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:12.117014 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:12.117054 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:12.117137 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:12.117504 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:12.117514 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:12.117522 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.117720 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.118134 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":268,"threads_started":1}
I20260926 07:32:12.120824 21730 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38119
I20260926 07:32:12.121155 21730 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:12.121922 21730 heartbeater.cc:511] Master 127.2.147.254:38119 requested a full tablet report, sending...
I20260926 07:32:12.123500 21413 ts_manager.cc:194] Registered new tserver with Master: d05b6c9b42bb43639b7ca9bc651935b1 (127.2.147.194:39571)
I20260926 07:32:12.124768 21413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.194:53561
W20260926 07:32:12.444923 21733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:12.445199 21733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:12.445361 21733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:12.452795 21733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:12.452895 21733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:12.453001 21733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:12.462016 21733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:12.463336 21733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:12.465091 21733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:12.478255 21738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:12.478282 21739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:12.478206 21741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:12.478982 21733 server_base.cc:1034] running on GCE node
I20260926 07:32:12.479628 21733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:12.480573 21733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:12.481820 21733 hybrid_clock.cc:648] HybridClock initialized: now 1790407932481771 us; error 53 us; skew 500 ppm
I20260926 07:32:12.482158 21733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:12.484464 21733 webserver.cc:460] Webserver started at http://127.2.147.195:45053/ using document root <none> and password file <none>
I20260926 07:32:12.485172 21733 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:12.485301 21733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:12.485627 21733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:12.488586 21733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "f633843308334da1af3f6dd7ea32b71c"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "0bd1478101b27b0d7c678bc67878e97a"
server_key_iv: "1f844c18504de2af4d0e2ca718cae2cf"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.489389 21733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "f633843308334da1af3f6dd7ea32b71c"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "0bd1478101b27b0d7c678bc67878e97a"
server_key_iv: "1f844c18504de2af4d0e2ca718cae2cf"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.495014 21733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:32:12.498718 21747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.500182 21733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:12.500363 21733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "f633843308334da1af3f6dd7ea32b71c"
format_stamp: "Formatted at 2026-09-26 07:32:12 on dist-test-slave-1dwd"
server_key: "0bd1478101b27b0d7c678bc67878e97a"
server_key_iv: "1f844c18504de2af4d0e2ca718cae2cf"
server_key_version: "encryptionkey@0"
I20260926 07:32:12.500604 21733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:12.515539 21733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:12.516480 21733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:12.516781 21733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:12.517793 21733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:12.519268 21733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:12.519380 21733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.519511 21733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:12.519598 21733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:12.552109 21733 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:35467
I20260926 07:32:12.552147 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:35467 every 8 connection(s)
I20260926 07:32:12.553444 21733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:12.563565 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21733
I20260926 07:32:12.563746 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:12.564288 2639 external_mini_cluster.cc:1428] Setting key 21fb6dab2b985127564da1ec5252c350
I20260926 07:32:12.565869 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.556033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:43733 (local address 127.2.147.254:38119)
0926 07:32:12.556205 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.556209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.557539 (+ 1330us) server_negotiation.cc:407] Connection header received
0926 07:32:12.559276 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.559288 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.559358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.559445 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:12.560481 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.561229 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.562695 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.563083 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.563180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.564792 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:12.564798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:12.564802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:12.564826 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:12.564864 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:12.564881 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:12.564978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.565467 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.565575 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:32:12.566946 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.556507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38119 (local address 127.2.147.195:43733)
0926 07:32:12.557360 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.557400 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.558990 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.559645 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.559653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.559734 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:12.560330 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.560345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.561358 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.561362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.562574 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.562586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.563332 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.563340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.563649 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.564418 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:12.564421 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:12.564472 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:12.564567 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:12.565214 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:12.565222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:12.565228 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.565429 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.565837 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":267,"threads_started":1}
I20260926 07:32:12.568284 21860 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38119
I20260926 07:32:12.568531 21860 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:12.569216 21860 heartbeater.cc:511] Master 127.2.147.254:38119 requested a full tablet report, sending...
I20260926 07:32:12.570510 21413 ts_manager.cc:194] Registered new tserver with Master: f633843308334da1af3f6dd7ea32b71c (127.2.147.195:35467)
I20260926 07:32:12.571398 21413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.195:43733
I20260926 07:32:12.579473 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:12.590684 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.583534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.2.147.254:38119)
0926 07:32:12.583769 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.583779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.584108 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:32:12.584236 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.584240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.584304 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.584385 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:12.584798 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.585497 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.586958 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.587608 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.587727 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.589682 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:12.589688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:12.589699 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:12.589738 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:12.589778 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:12.589793 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:12.590056 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.590289 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.590356 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:12.595265 21413 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:12.602557 21413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:12.628567 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.615255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35467 (local address 127.0.0.1:60356)
0926 07:32:12.616259 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.616303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.616480 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.618759 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.618770 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.618814 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:12.619984 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.620004 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.622716 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.622722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.625428 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.625442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.627552 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.627558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.627768 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.627831 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.627978 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.628196 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":472,"threads_started":1}
I20260926 07:32:12.628556 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.615646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.2.147.195:35467)
0926 07:32:12.617946 (+ 2300us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.617959 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.617999 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:12.618127 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.618139 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.618458 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.618716 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:12.620274 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.622513 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.625649 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.627316 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.627470 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.627776 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.628049 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.628220 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2069,"thread_start_us":2246,"threads_started":1}
I20260926 07:32:12.631978 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.615671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:39571 (local address 127.0.0.1:35648)
0926 07:32:12.617078 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.617092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.617201 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.624573 (+ 7372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.624578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.624604 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:12.625001 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.625018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.627049 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.627055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.629192 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.629206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.631113 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.631120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.631338 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.631371 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.631482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.631619 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":1121,"threads_started":1}
I20260926 07:32:12.632227 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.616305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.2.147.194:39571)
0926 07:32:12.623888 (+ 7583us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.623897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.623932 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:32:12.624041 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.624048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.624350 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.624573 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:12.625191 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.626886 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.629418 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.630873 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.631026 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.631287 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.631537 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.631711 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7436,"thread_start_us":7332,"threads_started":1}
I20260926 07:32:12.633137 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.616784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:33433 (local address 127.0.0.1:48268)
0926 07:32:12.623117 (+ 6333us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.623142 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.623359 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.624073 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.624078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.624103 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:12.624558 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.624572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.626427 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.626435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.628463 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.628478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.631892 (+ 3414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.631898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.632489 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.632543 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.632639 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.632767 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6146,"thread_start_us":5912,"threads_started":1}
I20260926 07:32:12.633342 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.617389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.2.147.193:33433)
0926 07:32:12.619734 (+ 2345us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.619746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.623307 (+ 3561us) server_negotiation.cc:407] Connection header received
0926 07:32:12.623550 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.623559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.623847 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.624068 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:12.624794 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.626286 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.629023 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.631317 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.631507 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.631738 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.632709 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.632878 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2121,"thread_start_us":1953,"threads_started":1}
I20260926 07:32:12.633090 21795 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 (DEFAULT_TABLE table=test-table [id=8984eb0b0a474209a51e8b367d3baca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:12.635974 21665 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 (DEFAULT_TABLE table=test-table [id=8984eb0b0a474209a51e8b367d3baca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:12.636747 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d419636451ea6bdf68ed91c4027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:12.637132 21665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d419636451ea6bdf68ed91c4027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:12.636861 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 (DEFAULT_TABLE table=test-table [id=8984eb0b0a474209a51e8b367d3baca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:12.638415 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bf1d419636451ea6bdf68ed91c4027. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:12.655267 21878 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Bootstrap starting.
I20260926 07:32:12.659030 21878 tablet_bootstrap.cc:654] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:12.660173 21878 log.cc:824] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:12.660252 21879 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Bootstrap starting.
I20260926 07:32:12.662506 21880 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Bootstrap starting.
I20260926 07:32:12.664543 21879 tablet_bootstrap.cc:654] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:12.665943 21879 log.cc:824] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:12.667073 21880 tablet_bootstrap.cc:654] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:12.668169 21599 heartbeater.cc:503] Master 127.2.147.254:38119 was elected leader, sending a full tablet report...
I20260926 07:32:12.668372 21880 log.cc:824] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:12.668431 21878 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: No bootstrap required, opened a new log
I20260926 07:32:12.668915 21878 ts_tablet_manager.cc:1397] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260926 07:32:12.675540 21879 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: No bootstrap required, opened a new log
I20260926 07:32:12.675882 21879 ts_tablet_manager.cc:1397] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260926 07:32:12.676658 21880 tablet_bootstrap.cc:492] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: No bootstrap required, opened a new log
I20260926 07:32:12.676949 21880 ts_tablet_manager.cc:1397] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 07:32:12.681547 21878 raft_consensus.cc:348] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.682098 21878 raft_consensus.cc:374] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:12.682201 21878 raft_consensus.cc:729] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05b6c9b42bb43639b7ca9bc651935b1, State: Initialized, Role: FOLLOWER
I20260926 07:32:12.682866 21878 consensus_queue.cc:260] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.684067 21730 heartbeater.cc:503] Master 127.2.147.254:38119 was elected leader, sending a full tablet report...
I20260926 07:32:12.684720 21878 ts_tablet_manager.cc:1428] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260926 07:32:12.685124 21880 raft_consensus.cc:348] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.685559 21880 raft_consensus.cc:374] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:12.685676 21880 raft_consensus.cc:729] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f633843308334da1af3f6dd7ea32b71c, State: Initialized, Role: FOLLOWER
I20260926 07:32:12.686292 21880 consensus_queue.cc:260] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.687542 21860 heartbeater.cc:503] Master 127.2.147.254:38119 was elected leader, sending a full tablet report...
I20260926 07:32:12.687767 21879 raft_consensus.cc:348] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.688213 21880 ts_tablet_manager.cc:1428] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 07:32:12.688225 21879 raft_consensus.cc:374] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:12.688324 21879 raft_consensus.cc:729] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a184907958ab4730b830d6fd47a71b43, State: Initialized, Role: FOLLOWER
I20260926 07:32:12.688931 21879 consensus_queue.cc:260] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.690802 21879 ts_tablet_manager.cc:1428] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 07:32:12.703666 21886 raft_consensus.cc:482] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:12.703997 21886 raft_consensus.cc:504] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.705786 21886 leader_election.cc:288] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f633843308334da1af3f6dd7ea32b71c (127.2.147.195:35467), d05b6c9b42bb43639b7ca9bc651935b1 (127.2.147.194:39571)
I20260926 07:32:12.713845 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.706246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:50807 (local address 127.2.147.194:39571)
0926 07:32:12.706474 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.706483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.707305 (+ 822us) server_negotiation.cc:407] Connection header received
0926 07:32:12.707582 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.707586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.707695 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.707790 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:12.708741 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.709780 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.712077 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.712842 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.713034 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.713106 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.713426 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.713569 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:32:12.713847 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.706384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:53375 (local address 127.2.147.195:35467)
0926 07:32:12.706595 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.706604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.707415 (+ 811us) server_negotiation.cc:407] Connection header received
0926 07:32:12.707670 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.707675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.707794 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.707912 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:12.708619 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.709762 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.712107 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.712888 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.713071 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.713141 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.713456 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.713596 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:32:12.714051 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.706269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:39571 (local address 127.2.147.193:50807)
0926 07:32:12.707154 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.707170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.707381 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.707962 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.707966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.708000 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:12.708525 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.708537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.709937 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.709942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.711865 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.711878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.713049 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.713055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.713227 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.713292 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.713397 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.713569 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":153,"thread_start_us":491,"threads_started":1}
I20260926 07:32:12.714308 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.706271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35467 (local address 127.2.147.193:53375)
0926 07:32:12.707295 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:12.707316 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:12.707466 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:12.707932 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:12.707937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:12.707966 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:12.708444 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.708453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.709981 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.710003 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.711869 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:12.711882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.713097 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.713103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:12.713282 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.713305 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:32:12.713404 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:32:12.713581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":108,"thread_start_us":473,"threads_started":1}
I20260926 07:32:12.715039 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d419636451ea6bdf68ed91c4027" candidate_uuid: "a184907958ab4730b830d6fd47a71b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" is_pre_election: true
I20260926 07:32:12.715284 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d419636451ea6bdf68ed91c4027" candidate_uuid: "a184907958ab4730b830d6fd47a71b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f633843308334da1af3f6dd7ea32b71c" is_pre_election: true
I20260926 07:32:12.715672 21685 raft_consensus.cc:2454] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a184907958ab4730b830d6fd47a71b43 in term 0.
I20260926 07:32:12.715772 21815 raft_consensus.cc:2454] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a184907958ab4730b830d6fd47a71b43 in term 0.
I20260926 07:32:12.716432 21489 leader_election.cc:302] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a184907958ab4730b830d6fd47a71b43, d05b6c9b42bb43639b7ca9bc651935b1, f633843308334da1af3f6dd7ea32b71c; no voters:
I20260926 07:32:12.716949 21886 raft_consensus.cc:2781] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:12.717144 21886 raft_consensus.cc:482] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:12.717255 21886 raft_consensus.cc:3037] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:12.719645 21886 raft_consensus.cc:504] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.720353 21886 leader_election.cc:288] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [CANDIDATE]: Term 1 election: Requested vote from peers f633843308334da1af3f6dd7ea32b71c (127.2.147.195:35467), d05b6c9b42bb43639b7ca9bc651935b1 (127.2.147.194:39571)
I20260926 07:32:12.720922 21685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d419636451ea6bdf68ed91c4027" candidate_uuid: "a184907958ab4730b830d6fd47a71b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05b6c9b42bb43639b7ca9bc651935b1"
I20260926 07:32:12.720884 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bf1d419636451ea6bdf68ed91c4027" candidate_uuid: "a184907958ab4730b830d6fd47a71b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f633843308334da1af3f6dd7ea32b71c"
I20260926 07:32:12.721139 21685 raft_consensus.cc:3037] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:12.721139 21815 raft_consensus.cc:3037] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:12.724568 21685 raft_consensus.cc:2454] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a184907958ab4730b830d6fd47a71b43 in term 1.
I20260926 07:32:12.724556 21815 raft_consensus.cc:2454] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a184907958ab4730b830d6fd47a71b43 in term 1.
I20260926 07:32:12.725061 21490 leader_election.cc:302] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a184907958ab4730b830d6fd47a71b43, f633843308334da1af3f6dd7ea32b71c; no voters:
I20260926 07:32:12.725385 21886 raft_consensus.cc:2781] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:12.725827 21886 raft_consensus.cc:686] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 1 LEADER]: Becoming Leader. State: Replica: a184907958ab4730b830d6fd47a71b43, State: Running, Role: LEADER
I20260926 07:32:12.726305 21886 consensus_queue.cc:237] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } }
I20260926 07:32:12.731031 21411 catalog_manager.cc:5495] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to a184907958ab4730b830d6fd47a71b43 (127.2.147.193). New cstate: current_term: 1 leader_uuid: "a184907958ab4730b830d6fd47a71b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a184907958ab4730b830d6fd47a71b43" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 33433 } health_report { overall_health: HEALTHY } } }
I20260926 07:32:12.766781 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.761078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.2.147.193:33433)
0926 07:32:12.761289 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.761296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.761381 (+ 85us) server_negotiation.cc:407] Connection header received
0926 07:32:12.761630 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.761634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.761724 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.761880 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:12.762274 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.762962 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.764415 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.764839 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.764980 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.765623 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:12.766134 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:12.766344 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.766465 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.766527 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:32:12.775126 21815 raft_consensus.cc:1261] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 1 FOLLOWER]: Refusing update from remote peer a184907958ab4730b830d6fd47a71b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:12.775125 21685 raft_consensus.cc:1261] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 1 FOLLOWER]: Refusing update from remote peer a184907958ab4730b830d6fd47a71b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:12.776201 21886 consensus_queue.cc:1035] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f633843308334da1af3f6dd7ea32b71c" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:12.776502 21889 consensus_queue.cc:1035] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d05b6c9b42bb43639b7ca9bc651935b1" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:12.791926 21892 mvcc.cc:205] Tried to move back new op lower bound from 7333510892636962816 to 7333510892450996224. Current Snapshot: MvccSnapshot[applied={T|T < 7333510892636962816}]
I20260926 07:32:12.805902 21411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:12.806252 21411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:12.809954 21411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5202a66b46448d18a75bb0914ffcd63: Sending DeleteTablet for 3 replicas of tablet 67bf1d419636451ea6bdf68ed91c4027
I20260926 07:32:12.811111 21795 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:12 UTC) from {username='slave'} at 127.0.0.1:60356
I20260926 07:32:12.811272 21665 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:12 UTC) from {username='slave'} at 127.0.0.1:35648
I20260926 07:32:12.811559 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bf1d419636451ea6bdf68ed91c4027 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:12 UTC) from {username='slave'} at 127.0.0.1:48268
I20260926 07:32:12.813210 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21472
I20260926 07:32:12.824421 21903 tablet_replica.cc:330] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: stopping tablet replica
I20260926 07:32:12.825234 21903 raft_consensus.cc:2229] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:12.826367 21903 raft_consensus.cc:2258] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:12.829731 21903 ts_tablet_manager.cc:1905] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:12.842024 21903 ts_tablet_manager.cc:1918] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:12.842419 21903 log.cc:1196] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/67bf1d419636451ea6bdf68ed91c4027
I20260926 07:32:12.843286 21903 ts_tablet_manager.cc:1939] T 67bf1d419636451ea6bdf68ed91c4027 P a184907958ab4730b830d6fd47a71b43: Deleting consensus metadata
I20260926 07:32:12.844182 21902 tablet_replica.cc:330] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: stopping tablet replica
I20260926 07:32:12.844673 21902 raft_consensus.cc:2229] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:12.845228 21902 raft_consensus.cc:2258] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:12.846343 21400 catalog_manager.cc:4841] TS a184907958ab4730b830d6fd47a71b43 (127.2.147.193:33433): tablet 67bf1d419636451ea6bdf68ed91c4027 (table test-table [id=8984eb0b0a474209a51e8b367d3baca1]) successfully deleted
I20260926 07:32:12.847610 21901 tablet_replica.cc:330] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: stopping tablet replica
I20260926 07:32:12.848186 21901 raft_consensus.cc:2229] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:12.848798 21901 raft_consensus.cc:2258] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:12.849818 21902 ts_tablet_manager.cc:1905] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:12.850190 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.816277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.2.147.193:33433)
0926 07:32:12.816449 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.816456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.838903 (+ 22447us) server_negotiation.cc:407] Connection header received
0926 07:32:12.839462 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.839468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.839588 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.839665 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:12.840423 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.841713 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.845727 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.846422 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.846529 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.848411 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:12.848422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:12.848440 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:12.849258 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:12.849322 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:12.849347 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:12.849432 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.849685 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.849792 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:32:12.851152 21901 ts_tablet_manager.cc:1905] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:12.851385 21594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:12.857645 21902 ts_tablet_manager.cc:1918] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:12.857839 21902 log.cc:1196] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/67bf1d419636451ea6bdf68ed91c4027
I20260926 07:32:12.858361 21902 ts_tablet_manager.cc:1939] T 67bf1d419636451ea6bdf68ed91c4027 P d05b6c9b42bb43639b7ca9bc651935b1: Deleting consensus metadata
I20260926 07:32:12.858832 21901 ts_tablet_manager.cc:1918] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:12.859002 21901 log.cc:1196] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/67bf1d419636451ea6bdf68ed91c4027
I20260926 07:32:12.859552 21901 ts_tablet_manager.cc:1939] T 67bf1d419636451ea6bdf68ed91c4027 P f633843308334da1af3f6dd7ea32b71c: Deleting consensus metadata
I20260926 07:32:12.860508 21399 catalog_manager.cc:4841] TS d05b6c9b42bb43639b7ca9bc651935b1 (127.2.147.194:39571): tablet 67bf1d419636451ea6bdf68ed91c4027 (table test-table [id=8984eb0b0a474209a51e8b367d3baca1]) successfully deleted
I20260926 07:32:12.861177 21400 catalog_manager.cc:4841] TS f633843308334da1af3f6dd7ea32b71c (127.2.147.195:35467): tablet 67bf1d419636451ea6bdf68ed91c4027 (table test-table [id=8984eb0b0a474209a51e8b367d3baca1]) successfully deleted
I20260926 07:32:12.952950 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21472
I20260926 07:32:12.971010 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21603
I20260926 07:32:12.977348 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:12.971879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.2.147.194:39571)
0926 07:32:12.972049 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:12.972053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:12.972072 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:32:12.972129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:12.972133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:12.972206 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:12.972291 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:12.972636 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.973351 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.974546 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:12.974925 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:12.975011 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:12.976151 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:12.976159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:12.976173 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:12.976759 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:12.976810 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:12.976829 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:12.976915 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:12.977030 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:32:12.977113 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:32:12.978018 21725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:13.067247 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21603
I20260926 07:32:13.084601 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21733
I20260926 07:32:13.091606 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:13.085405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.2.147.195:35467)
0926 07:32:13.085594 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:13.085600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:13.085620 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:32:13.085791 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:13.085797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:13.085901 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:13.086023 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:13.086412 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:13.087185 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:13.088476 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:13.088869 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:13.089007 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:13.090129 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:13.090138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:13.090154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:13.090704 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:13.090750 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:13.090768 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:13.090915 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:13.091055 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:32:13.091164 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:13.092243 21855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:13.182039 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21733
I20260926 07:32:13.199185 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21382
I20260926 07:32:13.200438 21443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:13.309830 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21382
2026-09-26T07:32:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2744 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:13 dist-test-slave-1dwd krb5kdc[21911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:13 dist-test-slave-1dwd krb5kdc[21911](info): set up 2 sockets
Sep 26 07:32:13 dist-test-slave-1dwd krb5kdc[21911](info): commencing operation
krb5kdc: starting...
W20260926 07:32:14.427646 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:14 dist-test-slave-1dwd krb5kdc[21911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:14.679374 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38793
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:15.008639 21924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:15.008931 21924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:15.009140 21924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:15.016616 21924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:15.016719 21924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:15.016824 21924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:15.016925 21924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:15.017004 21924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:15.027297 21924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:15.028654 21924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:15.030045 21924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:15.042057 21929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:15.042451 21930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:15.042508 21932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:15.042810 21924 server_base.cc:1034] running on GCE node
I20260926 07:32:15.043671 21924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:15.045059 21924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:15.046294 21924 hybrid_clock.cc:648] HybridClock initialized: now 1790407935046249 us; error 38 us; skew 500 ppm
I20260926 07:32:15.046646 21924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:15 dist-test-slave-1dwd krb5kdc[21911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407935, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:15.049759 21924 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:15.051342 21924 webserver.cc:460] Webserver started at http://127.2.147.254:46591/ using document root <none> and password file <none>
I20260926 07:32:15.052062 21924 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:15.052206 21924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:15.052527 21924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:15.055526 21924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "e6532ccc00ac4f80972af60f3bab9e14"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "87c83d266448b58709fcfa2d067c79b5"
server_key_iv: "8d523e2b9e5ccf7091584bae491914c6"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.056273 21924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "e6532ccc00ac4f80972af60f3bab9e14"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "87c83d266448b58709fcfa2d067c79b5"
server_key_iv: "8d523e2b9e5ccf7091584bae491914c6"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.061398 21924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:15.064800 21939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:15.066166 21924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:15.066354 21924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "e6532ccc00ac4f80972af60f3bab9e14"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "87c83d266448b58709fcfa2d067c79b5"
server_key_iv: "8d523e2b9e5ccf7091584bae491914c6"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.066555 21924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:15.086544 21924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:15.092765 21924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:15.093098 21924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:15.115049 21924 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38793
I20260926 07:32:15.115062 21990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38793 every 8 connection(s)
I20260926 07:32:15.116828 21924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:15.122298 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:15.123787 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 21924
I20260926 07:32:15.124014 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:15.124445 2639 external_mini_cluster.cc:1428] Setting key ade2170c4e629fad23d6d0072c56539f
I20260926 07:32:15.135746 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: Bootstrap starting.
Sep 26 07:32:15 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:15.138758 21991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:15.139922 21991 log.cc:824] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:15.142872 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: No bootstrap required, opened a new log
I20260926 07:32:15.146945 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:15.126474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.2.147.254:38793)
0926 07:32:15.127355 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:15.127371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:15.127425 (+ 54us) server_negotiation.cc:407] Connection header received
0926 07:32:15.129326 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:15.129351 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:15.129753 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:15.130187 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:15.131696 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.132995 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.134317 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.134983 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.135165 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:15.139175 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:15.139183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:15.139196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:15.139289 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:15.142381 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:15.143129 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:15.143138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:15.143145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:15.143401 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:15.143779 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:15.143784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:15.143787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:15.144464 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:15.144786 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:15.145152 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:32:15.145607 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":343,"threads_started":1}
I20260926 07:32:15.152038 21991 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } }
I20260926 07:32:15.152417 21991 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:15.152493 21991 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6532ccc00ac4f80972af60f3bab9e14, State: Initialized, Role: FOLLOWER
I20260926 07:32:15.152969 21991 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } }
I20260926 07:32:15.153169 21991 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:15.153265 21991 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:15.153378 21991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:15.155702 21991 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } }
I20260926 07:32:15.156219 21991 leader_election.cc:302] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6532ccc00ac4f80972af60f3bab9e14; no voters:
I20260926 07:32:15.156843 21991 leader_election.cc:288] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:15.156985 21996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:15.157677 21996 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [term 1 LEADER]: Becoming Leader. State: Replica: e6532ccc00ac4f80972af60f3bab9e14, State: Running, Role: LEADER
I20260926 07:32:15.158175 21996 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } }
I20260926 07:32:15.158872 21991 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:15.161350 21997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6532ccc00ac4f80972af60f3bab9e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } } }
I20260926 07:32:15.161801 21997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:15.161384 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6532ccc00ac4f80972af60f3bab9e14. Latest consensus state: current_term: 1 leader_uuid: "e6532ccc00ac4f80972af60f3bab9e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6532ccc00ac4f80972af60f3bab9e14" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38793 } } }
I20260926 07:32:15.162461 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:15.164057 22003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:15.169463 22003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:15.178530 22003 catalog_manager.cc:1347] Generated new cluster ID: 337fb6cdd36e496ca2622eb4862cd7dc
I20260926 07:32:15.178661 22003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:15.210757 22003 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:15.212776 22003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:15.224403 22003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: Generated new TSK 0
I20260926 07:32:15.225476 22003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
Entry for principal kudu/127.2.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab.
I20260926 07:32:15.298672 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:15.630669 22017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:15.630988 22017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:15.631158 22017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:15.638553 22017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:15.638703 22017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:15.638870 22017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:15.648635 22017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.193.keytab
--principal=kudu/127.2.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:15.649955 22017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:15.651302 22017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:15.664116 22022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:15.664316 22023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:15.664652 22025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:15.665182 22017 server_base.cc:1034] running on GCE node
I20260926 07:32:15.665815 22017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:15.666706 22017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:15.667932 22017 hybrid_clock.cc:648] HybridClock initialized: now 1790407935667888 us; error 52 us; skew 500 ppm
I20260926 07:32:15.668241 22017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:15 dist-test-slave-1dwd krb5kdc[21911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407935, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:15.671566 22017 init.cc:377] Logged in from keytab as kudu/127.2.147.193@KRBTEST.COM (short username kudu)
I20260926 07:32:15.673034 22017 webserver.cc:460] Webserver started at http://127.2.147.193:38383/ using document root <none> and password file <none>
I20260926 07:32:15.673698 22017 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:15.673842 22017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:15.674220 22017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:15.677067 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "3002513f55d84074965cc328dbe71e6b"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "150c12acb245e43a871a22f676f877cf"
server_key_iv: "d9a8ff98cf282eca08ad2de8571e7c28"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.677856 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "3002513f55d84074965cc328dbe71e6b"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "150c12acb245e43a871a22f676f877cf"
server_key_iv: "d9a8ff98cf282eca08ad2de8571e7c28"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.682860 22017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:32:15.686337 22032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:15.687750 22017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:15.687933 22017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "3002513f55d84074965cc328dbe71e6b"
format_stamp: "Formatted at 2026-09-26 07:32:15 on dist-test-slave-1dwd"
server_key: "150c12acb245e43a871a22f676f877cf"
server_key_iv: "d9a8ff98cf282eca08ad2de8571e7c28"
server_key_version: "encryptionkey@0"
I20260926 07:32:15.688133 22017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:15.702443 22017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:15.706705 22017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:15.707000 22017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:15.708007 22017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:15.709380 22017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:15.709491 22017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:15.709661 22017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:15.709741 22017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:15.742425 22017 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:42043
I20260926 07:32:15.742465 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:42043 every 8 connection(s)
I20260926 07:32:15.743793 22017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:15.745450 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22017
I20260926 07:32:15.745636 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:15.746196 2639 external_mini_cluster.cc:1428] Setting key 3f263886986fce10ad3008dc5cd25de5
Sep 26 07:32:15 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407935, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.193@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:15.766512 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:15.746444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:60065 (local address 127.2.147.254:38793)
0926 07:32:15.747178 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:15.747184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:15.747984 (+ 800us) server_negotiation.cc:407] Connection header received
0926 07:32:15.749860 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:15.749865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:15.749945 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:15.750075 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:15.751612 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.752798 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.754985 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.755760 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.755866 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:15.759496 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:15.759505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:15.759512 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:15.759556 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:15.763050 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:15.763881 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:15.763889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:15.763892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:15.764043 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:15.764464 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:15.764472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:15.764478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:15.764885 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:15.765033 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:15.765780 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 07:32:15.766030 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":241,"threads_started":1}
I20260926 07:32:15.767064 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:15.747032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38793 (local address 127.2.147.193:60065)
0926 07:32:15.747841 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:15.747878 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:15.749608 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:15.750293 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:15.750301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:15.750800 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:15.751425 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:15.751450 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.752976 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.752984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:15.754829 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:15.754844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:15.756000 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:15.756008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:15.756326 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:15.757067 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:15.757088 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:15.759272 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:15.763190 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:15.763196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:15.763259 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:15.763755 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:15.764157 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:15.764160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:15.764162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:15.764340 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:15.765006 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:15.765011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:15.765376 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 07:32:15.765608 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 07:32:15.765974 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":359,"threads_started":1}
I20260926 07:32:15.769054 22145 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38793
I20260926 07:32:15.769367 22145 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:15.770162 22145 heartbeater.cc:511] Master 127.2.147.254:38793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.147.194@KRBTEST.COM; defaulting to no policy
I20260926 07:32:15.772421 21956 ts_manager.cc:194] Registered new tserver with Master: 3002513f55d84074965cc328dbe71e6b (127.2.147.193:42043)
I20260926 07:32:15.773881 21956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.193@KRBTEST.COM'} at 127.2.147.193:60065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
Entry for principal kudu/127.2.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab.
I20260926 07:32:15.803294 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:16.128854 22151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:16.129127 22151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:16.129221 22151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:16.136812 22151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:16.136924 22151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:16.137032 22151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:16.146638 22151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.194.keytab
--principal=kudu/127.2.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:16.147847 22151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:16.149048 22151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:16.162123 22159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:16.162168 22157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:16.162603 22156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:16.162762 22151 server_base.cc:1034] running on GCE node
I20260926 07:32:16.163491 22151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:16.164430 22151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:16.165652 22151 hybrid_clock.cc:648] HybridClock initialized: now 1790407936165618 us; error 33 us; skew 500 ppm
I20260926 07:32:16.165971 22151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:16 dist-test-slave-1dwd krb5kdc[21911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:16.172596 22151 init.cc:377] Logged in from keytab as kudu/127.2.147.194@KRBTEST.COM (short username kudu)
I20260926 07:32:16.174078 22151 webserver.cc:460] Webserver started at http://127.2.147.194:44629/ using document root <none> and password file <none>
I20260926 07:32:16.174768 22151 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:16.174901 22151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:16.175295 22151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:16.178174 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "2dcea56d171642cca42120f620f2279a"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "a6fc42c85b56cef5199b91354f369cc9"
server_key_iv: "4bed9a95f5867300614dc07567f85060"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.179186 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "2dcea56d171642cca42120f620f2279a"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "a6fc42c85b56cef5199b91354f369cc9"
server_key_iv: "4bed9a95f5867300614dc07567f85060"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.184383 22151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:32:16.187757 22166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.189054 22151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:32:16.189226 22151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "2dcea56d171642cca42120f620f2279a"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "a6fc42c85b56cef5199b91354f369cc9"
server_key_iv: "4bed9a95f5867300614dc07567f85060"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.189410 22151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:16.222296 22151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:16.227031 22151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:16.227357 22151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:16.228335 22151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:16.229673 22151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:16.229779 22151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.229938 22151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:16.230022 22151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.262737 22151 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:43327
I20260926 07:32:16.262822 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:43327 every 8 connection(s)
I20260926 07:32:16.264243 22151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:16.267398 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22151
I20260926 07:32:16.267575 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:16.268059 2639 external_mini_cluster.cc:1428] Setting key 8cd668e2717ce4df33b1bb1f651cb6e3
Sep 26 07:32:16 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.194@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:16.285590 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.266773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:43807 (local address 127.2.147.254:38793)
0926 07:32:16.266961 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.266965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.268395 (+ 1430us) server_negotiation.cc:407] Connection header received
0926 07:32:16.270600 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.270606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.270737 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.270849 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:16.272455 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.273563 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.275131 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.275883 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.276054 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.279615 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:16.279620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:16.279623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:16.279655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:16.282345 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.283314 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.283319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.283321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.283441 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.283907 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.283910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.283912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.284169 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:16.284276 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.285168 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.285344 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:16.287179 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.267348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38793 (local address 127.2.147.194:43807)
0926 07:32:16.268218 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.268269 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.270332 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.271118 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.271128 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.271698 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:16.272277 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.272294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.273764 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.273767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.274977 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.274999 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.276119 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.276123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.276318 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.277034 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:16.277060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:16.279403 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:16.282484 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:16.282495 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:16.282515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:16.283160 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:16.283548 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:16.283555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:16.283562 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:16.283795 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:16.284299 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:16.284310 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:16.284790 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.285100 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.285682 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":394,"threads_started":1}
I20260926 07:32:16.288616 22279 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38793
I20260926 07:32:16.288954 22279 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:16.289667 22279 heartbeater.cc:511] Master 127.2.147.254:38793 requested a full tablet report, sending...
I20260926 07:32:16.291253 21956 ts_manager.cc:194] Registered new tserver with Master: 2dcea56d171642cca42120f620f2279a (127.2.147.194:43327)
I20260926 07:32:16.292142 21956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.194@KRBTEST.COM'} at 127.2.147.194:43807
WARNING: no policy specified for kudu/127.2.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
Entry for principal kudu/127.2.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab.
I20260926 07:32:16.326092 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:16.658179 22284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:16.658502 22284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:16.658675 22284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:16.666143 22284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:16.666292 22284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:16.666484 22284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:16.676357 22284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.195.keytab
--principal=kudu/127.2.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:16.677673 22284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:16.679039 22284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:16.691730 22292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:16.691779 22289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:16.691882 22290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:16.692469 22284 server_base.cc:1034] running on GCE node
I20260926 07:32:16.693118 22284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:16.694152 22284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:16.695379 22284 hybrid_clock.cc:648] HybridClock initialized: now 1790407936695318 us; error 38 us; skew 500 ppm
I20260926 07:32:16.695761 22284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:16 dist-test-slave-1dwd krb5kdc[21911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:16.699144 22284 init.cc:377] Logged in from keytab as kudu/127.2.147.195@KRBTEST.COM (short username kudu)
I20260926 07:32:16.700557 22284 webserver.cc:460] Webserver started at http://127.2.147.195:40183/ using document root <none> and password file <none>
I20260926 07:32:16.701239 22284 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:16.701370 22284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:16.701687 22284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:16.704633 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "ecddf77d97ab418495d64972a89b1b6d"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "b36c76827a27a3b426b75de85ba5b6e6"
server_key_iv: "7e595dd81f34ce5dbb71eec225976d48"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.705461 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "ecddf77d97ab418495d64972a89b1b6d"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "b36c76827a27a3b426b75de85ba5b6e6"
server_key_iv: "7e595dd81f34ce5dbb71eec225976d48"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.710724 22284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:32:16.714075 22299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.715359 22284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:16.715521 22284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "ecddf77d97ab418495d64972a89b1b6d"
format_stamp: "Formatted at 2026-09-26 07:32:16 on dist-test-slave-1dwd"
server_key: "b36c76827a27a3b426b75de85ba5b6e6"
server_key_iv: "7e595dd81f34ce5dbb71eec225976d48"
server_key_version: "encryptionkey@0"
I20260926 07:32:16.715718 22284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:16.739427 22284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:16.744108 22284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:16.744387 22284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:16.745368 22284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:16.746743 22284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:16.746845 22284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.746999 22284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:16.747071 22284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:16.776937 22145 heartbeater.cc:503] Master 127.2.147.254:38793 was elected leader, sending a full tablet report...
I20260926 07:32:16.779274 22284 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:35619
I20260926 07:32:16.779332 22411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:35619 every 8 connection(s)
I20260926 07:32:16.780494 22284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:16.789768 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22284
I20260926 07:32:16.789942 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:16.790373 2639 external_mini_cluster.cc:1428] Setting key 99465ca8500d899e0c9d77c2718f9ccc
Sep 26 07:32:16 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407936, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.195@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:16.799397 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.782942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:60005 (local address 127.2.147.254:38793)
0926 07:32:16.783155 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.783161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.784418 (+ 1257us) server_negotiation.cc:407] Connection header received
0926 07:32:16.786148 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.786152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.786215 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.786302 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:16.787768 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.788521 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.789956 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.790432 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.790574 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.794495 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:16.794499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:16.794502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:16.794532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:16.796890 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.797523 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.797527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.797529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.797651 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.798002 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.798005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.798007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.798256 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:16.798359 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.798918 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.799070 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:32:16.800248 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.783559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38793 (local address 127.2.147.195:60005)
0926 07:32:16.784255 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.784290 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.785903 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.786507 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.786515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.786994 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:16.787626 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.787644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.788643 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.788646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.789804 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.789815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.790676 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.790684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.790961 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.791694 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:16.791716 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:16.794308 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:16.797013 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:16.797019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:16.797030 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:16.797420 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:16.797738 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:16.797741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:16.797743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:16.797907 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:16.798354 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:16.798359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:16.798622 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.798822 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.799181 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":263,"threads_started":1}
I20260926 07:32:16.801430 22412 heartbeater.cc:347] Connected to a master server at 127.2.147.254:38793
I20260926 07:32:16.801712 22412 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:16.802381 22412 heartbeater.cc:511] Master 127.2.147.254:38793 requested a full tablet report, sending...
I20260926 07:32:16.803798 21956 ts_manager.cc:194] Registered new tserver with Master: ecddf77d97ab418495d64972a89b1b6d (127.2.147.195:35619)
I20260926 07:32:16.804752 21956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.147.195@KRBTEST.COM'} at 127.2.147.195:60005
I20260926 07:32:16.805496 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:16.823633 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.809689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.2.147.254:38793)
0926 07:32:16.809880 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.809889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.810388 (+ 499us) server_negotiation.cc:407] Connection header received
0926 07:32:16.810562 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.810566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.810622 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.810698 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:16.812765 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.813460 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.815529 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.815905 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.815998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.818276 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:16.818280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:16.818283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:16.818311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:16.821122 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.821856 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.821860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.821862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.821988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:16.822305 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:16.822308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:16.822310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:16.822607 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:16.822782 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.823048 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.823287 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:32:16.827539 21955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:16.834368 21955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:16.862102 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.847357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.2.147.194:43327)
0926 07:32:16.853982 (+ 6625us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.854002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.854034 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:32:16.854144 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.854152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.854384 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.855426 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:16.855673 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.856827 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.859614 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.860930 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.861103 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.861315 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.861595 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.861758 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6458,"thread_start_us":6344,"threads_started":1}
I20260926 07:32:16.862185 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.846060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:43327 (local address 127.0.0.1:45600)
0926 07:32:16.852794 (+ 6734us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.852818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.852974 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.854595 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.854603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.854628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:16.855089 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.855102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.856962 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.856970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.859448 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.859464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.861144 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.861149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.861347 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.861399 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.861538 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.861736 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6573,"mutex_wait_us":1,"thread_start_us":1165,"threads_started":1}
I20260926 07:32:16.862948 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.845784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:42043 (local address 127.0.0.1:40916)
0926 07:32:16.846762 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.846813 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.847009 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.854514 (+ 7505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.854522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.854557 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:16.855512 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.855530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.857595 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.857600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.859894 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.859912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.861808 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.861814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.862024 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.862069 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.862138 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.862528 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":426,"spinlock_wait_cycles":6016,"thread_start_us":8239,"threads_started":1}
I20260926 07:32:16.863276 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.846960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.2.147.195:35619)
0926 07:32:16.852283 (+ 5323us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.852295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.854381 (+ 2086us) server_negotiation.cc:407] Connection header received
0926 07:32:16.854677 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.854687 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.854894 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.855068 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:16.855950 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.857586 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.860192 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.861566 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.861701 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.861941 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.862727 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.862898 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5132,"thread_start_us":4977,"threads_started":1}
I20260926 07:32:16.863658 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.846863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35619 (local address 127.0.0.1:41092)
0926 07:32:16.854272 (+ 7409us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.854288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.854481 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.855290 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.855299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.855319 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:16.855699 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.855710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.857724 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.857730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.859963 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.859978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.862082 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.862088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.862515 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.862549 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.862638 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.862753 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7289,"mutex_wait_us":229,"thread_start_us":5595,"threads_started":1}
I20260926 07:32:16.864507 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.845878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.2.147.193:42043)
0926 07:32:16.853748 (+ 7870us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.853760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.853798 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:32:16.853931 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.853942 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.854221 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.854934 (+ 713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:16.855719 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.857322 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.860062 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.861557 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.863555 (+ 1998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.863875 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.863945 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.864137 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7651,"thread_start_us":7455,"threads_started":1}
I20260926 07:32:16.866405 22214 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b (DEFAULT_TABLE table=test-table [id=714b64f0072348e9ad8e9e49cee82f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:16.867353 22080 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b (DEFAULT_TABLE table=test-table [id=714b64f0072348e9ad8e9e49cee82f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:16.867970 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6589f4fd5e48ebb6214429aa65dc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:16.867559 22347 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b (DEFAULT_TABLE table=test-table [id=714b64f0072348e9ad8e9e49cee82f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:16.868579 22080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6589f4fd5e48ebb6214429aa65dc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:16.868881 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6589f4fd5e48ebb6214429aa65dc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:16.880157 22430 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Bootstrap starting.
I20260926 07:32:16.882512 22432 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Bootstrap starting.
I20260926 07:32:16.883877 22431 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Bootstrap starting.
I20260926 07:32:16.883950 22430 tablet_bootstrap.cc:654] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:16.885277 22430 log.cc:824] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:16.887430 22431 tablet_bootstrap.cc:654] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:16.888725 22430 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: No bootstrap required, opened a new log
I20260926 07:32:16.888686 22431 log.cc:824] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:16.889009 22430 ts_tablet_manager.cc:1397] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:32:16.889145 22432 tablet_bootstrap.cc:654] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:16.890223 22432 log.cc:824] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:16.891878 22431 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: No bootstrap required, opened a new log
I20260926 07:32:16.892146 22431 ts_tablet_manager.cc:1397] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 07:32:16.893257 22432 tablet_bootstrap.cc:492] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: No bootstrap required, opened a new log
I20260926 07:32:16.893510 22432 ts_tablet_manager.cc:1397] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 07:32:16.897804 22430 raft_consensus.cc:348] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.898166 22430 raft_consensus.cc:374] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:16.898293 22430 raft_consensus.cc:729] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3002513f55d84074965cc328dbe71e6b, State: Initialized, Role: FOLLOWER
I20260926 07:32:16.898877 22430 consensus_queue.cc:260] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.900393 22430 ts_tablet_manager.cc:1428] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 07:32:16.902514 22431 raft_consensus.cc:348] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.902922 22431 raft_consensus.cc:374] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:16.903035 22431 raft_consensus.cc:729] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dcea56d171642cca42120f620f2279a, State: Initialized, Role: FOLLOWER
I20260926 07:32:16.903828 22431 consensus_queue.cc:260] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.904994 22279 heartbeater.cc:503] Master 127.2.147.254:38793 was elected leader, sending a full tablet report...
I20260926 07:32:16.905341 22432 raft_consensus.cc:348] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.905683 22431 ts_tablet_manager.cc:1428] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260926 07:32:16.905763 22432 raft_consensus.cc:374] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:16.905861 22432 raft_consensus.cc:729] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecddf77d97ab418495d64972a89b1b6d, State: Initialized, Role: FOLLOWER
I20260926 07:32:16.906483 22432 consensus_queue.cc:260] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.907666 22412 heartbeater.cc:503] Master 127.2.147.254:38793 was elected leader, sending a full tablet report...
I20260926 07:32:16.908303 22432 ts_tablet_manager.cc:1428] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 07:32:16.975003 22437 raft_consensus.cc:482] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:16.975363 22437 raft_consensus.cc:504] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.977038 22437 leader_election.cc:288] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3002513f55d84074965cc328dbe71e6b (127.2.147.193:42043), ecddf77d97ab418495d64972a89b1b6d (127.2.147.195:35619)
I20260926 07:32:16.983479 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.977441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:47901 (local address 127.2.147.195:35619)
0926 07:32:16.977649 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.977653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.978368 (+ 715us) server_negotiation.cc:407] Connection header received
0926 07:32:16.978618 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.978622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.978680 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.978774 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:16.979322 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.980057 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.981770 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.982532 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.982706 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.982786 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.983066 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.983192 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:32:16.983758 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.977399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:35619 (local address 127.2.147.194:47901)
0926 07:32:16.978255 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.978270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.978461 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.978791 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.978795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.978817 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:16.979152 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.979161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.980189 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.980197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.981608 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.981619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.982748 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.982754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.982899 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.982923 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.983039 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.983174 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":577,"threads_started":1}
I20260926 07:32:16.983887 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.977437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:36217 (local address 127.2.147.193:42043)
0926 07:32:16.977653 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:16.977658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:16.978868 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 07:32:16.979089 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:16.979094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:16.979158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:16.979293 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:16.979815 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.980513 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.982176 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.982931 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.983118 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.983190 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:16.983512 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:32:16.983638 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:32:16.984000 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:16.978017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:42043 (local address 127.2.147.194:36217)
0926 07:32:16.978770 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:16.978783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:16.978945 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:16.979306 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:16.979310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:16.979330 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:16.979641 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.979648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.980651 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.980658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.982021 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:16.982032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:16.983149 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:16.983155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:16.983355 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:16.983377 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:32:16.983479 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:32:16.983610 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":445,"threads_started":1}
I20260926 07:32:16.984820 22367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6589f4fd5e48ebb6214429aa65dc9b" candidate_uuid: "2dcea56d171642cca42120f620f2279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecddf77d97ab418495d64972a89b1b6d" is_pre_election: true
I20260926 07:32:16.984858 22100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6589f4fd5e48ebb6214429aa65dc9b" candidate_uuid: "2dcea56d171642cca42120f620f2279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3002513f55d84074965cc328dbe71e6b" is_pre_election: true
I20260926 07:32:16.985337 22367 raft_consensus.cc:2454] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcea56d171642cca42120f620f2279a in term 0.
I20260926 07:32:16.985350 22100 raft_consensus.cc:2454] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcea56d171642cca42120f620f2279a in term 0.
I20260926 07:32:16.985992 22170 leader_election.cc:302] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dcea56d171642cca42120f620f2279a, 3002513f55d84074965cc328dbe71e6b, ecddf77d97ab418495d64972a89b1b6d; no voters:
I20260926 07:32:16.986390 22437 raft_consensus.cc:2781] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:16.986538 22437 raft_consensus.cc:482] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:16.986640 22437 raft_consensus.cc:3037] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:16.988813 22437 raft_consensus.cc:504] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
I20260926 07:32:16.989418 22437 leader_election.cc:288] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [CANDIDATE]: Term 1 election: Requested vote from peers 3002513f55d84074965cc328dbe71e6b (127.2.147.193:42043), ecddf77d97ab418495d64972a89b1b6d (127.2.147.195:35619)
I20260926 07:32:16.989784 22100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6589f4fd5e48ebb6214429aa65dc9b" candidate_uuid: "2dcea56d171642cca42120f620f2279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3002513f55d84074965cc328dbe71e6b"
I20260926 07:32:16.989895 22367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6589f4fd5e48ebb6214429aa65dc9b" candidate_uuid: "2dcea56d171642cca42120f620f2279a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecddf77d97ab418495d64972a89b1b6d"
I20260926 07:32:16.989986 22100 raft_consensus.cc:3037] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:16.990130 22367 raft_consensus.cc:3037] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:16.992482 22367 raft_consensus.cc:2454] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcea56d171642cca42120f620f2279a in term 1.
I20260926 07:32:16.992555 22100 raft_consensus.cc:2454] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcea56d171642cca42120f620f2279a in term 1.
I20260926 07:32:16.992926 22169 leader_election.cc:302] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dcea56d171642cca42120f620f2279a, ecddf77d97ab418495d64972a89b1b6d; no voters:
I20260926 07:32:16.993268 22437 raft_consensus.cc:2781] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:16.993758 22437 raft_consensus.cc:686] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 1 LEADER]: Becoming Leader. State: Replica: 2dcea56d171642cca42120f620f2279a, State: Running, Role: LEADER
I20260926 07:32:16.994266 22437 consensus_queue.cc:237] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } }
W20260926 07:32:16.996106 22146 tablet.cc:2369] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:16.998833 21955 catalog_manager.cc:5495] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dcea56d171642cca42120f620f2279a (127.2.147.194). New cstate: current_term: 1 leader_uuid: "2dcea56d171642cca42120f620f2279a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dcea56d171642cca42120f620f2279a" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 } health_report { overall_health: UNKNOWN } } }
W20260926 07:32:17.016289 22280 tablet.cc:2369] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:32:17.032253 22413 tablet.cc:2369] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:17.047762 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:17.042827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.2.147.194:43327)
0926 07:32:17.043017 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:17.043021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:17.043077 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:32:17.043331 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:17.043335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:17.043422 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:17.043556 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:17.043891 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.044616 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.045887 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.046253 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.046341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:17.046759 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:17.047250 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:17.047352 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:17.047482 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:32:17.047552 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:32:17.055753 22100 raft_consensus.cc:1261] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 1 FOLLOWER]: Refusing update from remote peer 2dcea56d171642cca42120f620f2279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:17.055754 22367 raft_consensus.cc:1261] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 1 FOLLOWER]: Refusing update from remote peer 2dcea56d171642cca42120f620f2279a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:17.056787 22441 consensus_queue.cc:1035] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3002513f55d84074965cc328dbe71e6b" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 42043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:17.057134 22437 consensus_queue.cc:1035] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecddf77d97ab418495d64972a89b1b6d" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 35619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:17.066738 22444 mvcc.cc:205] Tried to move back new op lower bound from 7333510910170386432 to 7333510909932113920. Current Snapshot: MvccSnapshot[applied={T|T < 7333510910170386432}]
I20260926 07:32:17.068059 22445 mvcc.cc:205] Tried to move back new op lower bound from 7333510910170386432 to 7333510909932113920. Current Snapshot: MvccSnapshot[applied={T|T < 7333510910170386432}]
I20260926 07:32:17.085979 21955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:17.086339 21955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:17.090572 21955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6532ccc00ac4f80972af60f3bab9e14: Sending DeleteTablet for 3 replicas of tablet 0c6589f4fd5e48ebb6214429aa65dc9b
I20260926 07:32:17.091897 22080 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260926 07:32:17.092005 22346 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:41092
I20260926 07:32:17.092147 22347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:41092
I20260926 07:32:17.092815 22214 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6589f4fd5e48ebb6214429aa65dc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:17 UTC) from {username='kudu', principal='kudu/127.2.147.254@KRBTEST.COM'} at 127.0.0.1:45600
I20260926 07:32:17.094465 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22017
I20260926 07:32:17.133131 22454 tablet_replica.cc:330] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: stopping tablet replica
I20260926 07:32:17.133770 22454 raft_consensus.cc:2229] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:17.134420 22454 raft_consensus.cc:2258] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:17.139264 22456 tablet_replica.cc:330] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: stopping tablet replica
I20260926 07:32:17.139747 22456 raft_consensus.cc:2229] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:17.140358 22456 raft_consensus.cc:2258] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:17.143020 22455 tablet_replica.cc:330] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: stopping tablet replica
I20260926 07:32:17.143464 22455 raft_consensus.cc:2229] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:17.143977 22455 raft_consensus.cc:2258] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:17.145787 22454 ts_tablet_manager.cc:1905] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 07:32:17 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.193@KRBTEST.COM
I20260926 07:32:17.149029 22455 ts_tablet_manager.cc:1905] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:17.151507 22456 ts_tablet_manager.cc:1905] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:17.152384 22454 ts_tablet_manager.cc:1918] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:17.152561 22454 log.cc:1196] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/0c6589f4fd5e48ebb6214429aa65dc9b
I20260926 07:32:17.153117 22454 ts_tablet_manager.cc:1939] T 0c6589f4fd5e48ebb6214429aa65dc9b P 3002513f55d84074965cc328dbe71e6b: Deleting consensus metadata
I20260926 07:32:17.154981 21943 catalog_manager.cc:4841] TS 3002513f55d84074965cc328dbe71e6b (127.2.147.193:42043): tablet 0c6589f4fd5e48ebb6214429aa65dc9b (table test-table [id=714b64f0072348e9ad8e9e49cee82f52]) successfully deleted
I20260926 07:32:17.159497 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:17.096087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.2.147.193:42043)
0926 07:32:17.096312 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:17.096322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:17.129784 (+ 33462us) server_negotiation.cc:407] Connection header received
0926 07:32:17.130234 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:17.130241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:17.130324 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:17.130394 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:17.135095 (+ 4701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.136622 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.139256 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.140174 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.140302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:17.151424 (+ 11122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:17.151450 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:17.151468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:17.151520 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:17.155419 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.156254 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.156263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.156266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.156431 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.156889 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.156894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.156897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.157350 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:17.157642 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:17.158827 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0926 07:32:17.159055 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:17.159806 22455 ts_tablet_manager.cc:1918] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:17.160324 22140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:17.160656 22456 ts_tablet_manager.cc:1918] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:17.160845 22456 log.cc:1196] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/0c6589f4fd5e48ebb6214429aa65dc9b
I20260926 07:32:17.161320 22456 ts_tablet_manager.cc:1939] T 0c6589f4fd5e48ebb6214429aa65dc9b P 2dcea56d171642cca42120f620f2279a: Deleting consensus metadata
I20260926 07:32:17.162796 21940 catalog_manager.cc:4841] TS 2dcea56d171642cca42120f620f2279a (127.2.147.194:43327): tablet 0c6589f4fd5e48ebb6214429aa65dc9b (table test-table [id=714b64f0072348e9ad8e9e49cee82f52]) successfully deleted
I20260926 07:32:17.163324 22455 log.cc:1196] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/0c6589f4fd5e48ebb6214429aa65dc9b
I20260926 07:32:17.163806 22455 ts_tablet_manager.cc:1939] T 0c6589f4fd5e48ebb6214429aa65dc9b P ecddf77d97ab418495d64972a89b1b6d: Deleting consensus metadata
I20260926 07:32:17.165328 21942 catalog_manager.cc:4841] TS ecddf77d97ab418495d64972a89b1b6d (127.2.147.195:35619): tablet 0c6589f4fd5e48ebb6214429aa65dc9b (table test-table [id=714b64f0072348e9ad8e9e49cee82f52]) successfully deleted
W20260926 07:32:17.165923 21942 catalog_manager.cc:4808] TS ecddf77d97ab418495d64972a89b1b6d (127.2.147.195:35619): delete failed for tablet 0c6589f4fd5e48ebb6214429aa65dc9b because the tablet was not found. No further retry: Not found: Tablet not found: 0c6589f4fd5e48ebb6214429aa65dc9b
I20260926 07:32:17.253474 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22017
I20260926 07:32:17.272626 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22151
Sep 26 07:32:17 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.194@KRBTEST.COM
I20260926 07:32:17.288533 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:17.273402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.2.147.194:43327)
0926 07:32:17.273913 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:17.273917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:17.273932 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:32:17.273981 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:17.273985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:17.274085 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:17.274214 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:17.275547 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.276314 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.277572 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.278025 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.278169 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:17.282399 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:17.282411 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:17.282425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:17.282462 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:17.285559 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.286251 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.286258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.286261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.286412 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.286793 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.286798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.286802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.287288 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:17.287609 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:17.287885 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:32:17.288114 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260926 07:32:17.289211 22274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:17.404815 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22151
I20260926 07:32:17.424175 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22284
Sep 26 07:32:17 dist-test-slave-1dwd krb5kdc[21911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.195@KRBTEST.COM
I20260926 07:32:17.439065 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:17.425017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.2.147.195:35619)
0926 07:32:17.425178 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:17.425183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:17.425198 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:32:17.425254 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:17.425258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:17.425326 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:17.425428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:17.426710 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.427546 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.428932 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:17.429336 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:17.429433 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:17.433751 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:17.433764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:17.433782 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:17.433822 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:17.436515 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.437162 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.437169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.437171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.437326 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:17.437658 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:17.437661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:17.437663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:17.438034 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:17.438240 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:17.438504 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:32:17.438670 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:32:17.439703 22407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:17.528343 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22284
I20260926 07:32:17.545449 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 21924
I20260926 07:32:17.546576 21986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:17.662717 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 21924
2026-09-26T07:32:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:17Z Disabled control of system clock
I20260926 07:32:17.738600 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46051
--webserver_interface=127.2.147.254
--webserver_port=0
--builtin_ntp_servers=127.2.147.212:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:18.063448 22465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:18.063800 22465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:18.063944 22465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:18.071048 22465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:18.071195 22465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:18.071336 22465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:18.071441 22465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:18.071525 22465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:18.080572 22465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:18.081890 22465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:18.083734 22465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:18.095221 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:18.095252 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:18.095299 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:18.096022 22465 server_base.cc:1034] running on GCE node
I20260926 07:32:18.096722 22465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:18.097997 22465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:18.099273 22465 hybrid_clock.cc:648] HybridClock initialized: now 1790407938099192 us; error 49 us; skew 500 ppm
I20260926 07:32:18.099651 22465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:18.101759 22465 webserver.cc:460] Webserver started at http://127.2.147.254:33815/ using document root <none> and password file <none>
I20260926 07:32:18.102475 22465 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:18.102638 22465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:18.102968 22465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:18.106081 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "26f74f9d099340db9c95a7554b3f2a30"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "9999cd59890fc31b915ec323c1098433"
server_key_iv: "ee1a3e0072321d7c4af879bbdcf0caf4"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.106868 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "26f74f9d099340db9c95a7554b3f2a30"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "9999cd59890fc31b915ec323c1098433"
server_key_iv: "ee1a3e0072321d7c4af879bbdcf0caf4"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.112093 22465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 07:32:18.115653 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:18.121706 22465 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260926 07:32:18.121920 22465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "26f74f9d099340db9c95a7554b3f2a30"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "9999cd59890fc31b915ec323c1098433"
server_key_iv: "ee1a3e0072321d7c4af879bbdcf0caf4"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.122160 22465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:18.152882 22465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:18.153734 22465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:18.154013 22465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:18.171167 22465 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:46051
I20260926 07:32:18.171186 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:46051 every 8 connection(s)
I20260926 07:32:18.172514 22465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:18.176083 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:18.180965 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22465
I20260926 07:32:18.181133 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:18.181507 2639 external_mini_cluster.cc:1428] Setting key b3b3e773a325e931bb74e909eb23ae19
I20260926 07:32:18.187443 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: Bootstrap starting.
I20260926 07:32:18.191167 22531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:18.192342 22531 log.cc:824] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:18.195084 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: No bootstrap required, opened a new log
I20260926 07:32:18.196730 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:18.183533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.2.147.254:46051)
0926 07:32:18.184611 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:18.184627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:18.184677 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:32:18.186965 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:18.187007 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:18.187574 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:18.188014 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:18.188666 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.190653 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.192138 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.193181 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.193389 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:18.194505 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:18.194514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:18.194528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:18.195043 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:18.195122 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:18.195141 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:18.195285 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:18.195521 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:32:18.195841 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":321,"threads_started":1}
I20260926 07:32:18.202601 22531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } }
I20260926 07:32:18.202960 22531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:18.203069 22531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f74f9d099340db9c95a7554b3f2a30, State: Initialized, Role: FOLLOWER
I20260926 07:32:18.203644 22531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } }
I20260926 07:32:18.203889 22531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:18.204020 22531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:18.204198 22531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:18.206396 22531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } }
I20260926 07:32:18.206884 22531 leader_election.cc:302] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26f74f9d099340db9c95a7554b3f2a30; no voters:
I20260926 07:32:18.207461 22531 leader_election.cc:288] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:18.207600 22536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:18.208366 22536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [term 1 LEADER]: Becoming Leader. State: Replica: 26f74f9d099340db9c95a7554b3f2a30, State: Running, Role: LEADER
I20260926 07:32:18.208895 22536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } }
I20260926 07:32:18.209594 22531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:18.211723 22538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26f74f9d099340db9c95a7554b3f2a30. Latest consensus state: current_term: 1 leader_uuid: "26f74f9d099340db9c95a7554b3f2a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } } }
I20260926 07:32:18.212229 22538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:18.213195 22544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:18.213946 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26f74f9d099340db9c95a7554b3f2a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f74f9d099340db9c95a7554b3f2a30" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46051 } } }
I20260926 07:32:18.214270 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:18.218861 22544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:18.227988 22544 catalog_manager.cc:1347] Generated new cluster ID: ecccaaf3ac014976b07e41d8b0379360
I20260926 07:32:18.228116 22544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:18.242776 22544 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:18.244055 22544 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:18.251317 22544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: Generated new TSK 0
I20260926 07:32:18.252318 22544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:18.263465 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.193:0
--local_ip_for_outbound_sockets=127.2.147.193
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--builtin_ntp_servers=127.2.147.212:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:32:18.596649 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:18.596930 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:18.597119 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:18.604391 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:18.604506 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:18.604710 22555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.193
I20260926 07:32:18.613968 22555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.147.193
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.193
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:18.615531 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:18.617237 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:18.630366 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:18.630424 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:18.630458 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:18.631487 22555 server_base.cc:1034] running on GCE node
I20260926 07:32:18.632186 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:18.633117 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:18.634337 22555 hybrid_clock.cc:648] HybridClock initialized: now 1790407938634279 us; error 55 us; skew 500 ppm
I20260926 07:32:18.634650 22555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:18.636972 22555 webserver.cc:460] Webserver started at http://127.2.147.193:40417/ using document root <none> and password file <none>
I20260926 07:32:18.637687 22555 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:18.637826 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:18.638145 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:18.641011 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data/instance:
uuid: "edd83b13dc284706bee62411b9c5c689"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "4bcbcac6b813701868578c1e5aae7162"
server_key_iv: "5ce4ec5e007cb7d527ecd36eb45f149d"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.641770 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance:
uuid: "edd83b13dc284706bee62411b9c5c689"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "4bcbcac6b813701868578c1e5aae7162"
server_key_iv: "5ce4ec5e007cb7d527ecd36eb45f149d"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.646708 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:32:18.650267 22569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:18.651903 22555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:18.652081 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal
uuid: "edd83b13dc284706bee62411b9c5c689"
format_stamp: "Formatted at 2026-09-26 07:32:18 on dist-test-slave-1dwd"
server_key: "4bcbcac6b813701868578c1e5aae7162"
server_key_iv: "5ce4ec5e007cb7d527ecd36eb45f149d"
server_key_version: "encryptionkey@0"
I20260926 07:32:18.652297 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:18.669113 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:18.669919 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:18.670219 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:18.671160 22555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:18.672636 22555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:18.672771 22555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:18.672920 22555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:18.673012 22555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:18.705543 22555 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.193:41795
I20260926 07:32:18.705590 22681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.193:41795 every 8 connection(s)
I20260926 07:32:18.706840 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:18.711020 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22555
I20260926 07:32:18.711177 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:18.711741 2639 external_mini_cluster.cc:1428] Setting key 61e1e0ec92395a32427da63470845b48
I20260926 07:32:18.714497 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.194:0
--local_ip_for_outbound_sockets=127.2.147.194
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--builtin_ntp_servers=127.2.147.212:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:18.720369 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:18.709411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.193:33019 (local address 127.2.147.254:46051)
0926 07:32:18.709905 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:18.709910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:18.711439 (+ 1529us) server_negotiation.cc:407] Connection header received
0926 07:32:18.713792 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:18.713797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:18.713893 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:18.714011 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:18.715324 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.716041 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.717579 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.717980 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.718110 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:18.719565 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:18.719570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:18.719574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:18.719592 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:18.719662 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:18.719679 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:18.719782 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:18.720059 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:32:18.720136 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260926 07:32:18.721791 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:18.710109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46051 (local address 127.2.147.193:33019)
0926 07:32:18.711259 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:18.711308 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:18.713478 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:18.714275 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:18.714284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:18.714419 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:18.715141 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:18.715159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.716200 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.716205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:18.717439 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:18.717451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:18.718198 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:18.718208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:18.718464 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:18.719182 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:18.719184 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:18.719272 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:18.719360 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:18.719808 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:18.719822 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:18.719835 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:32:18.720024 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 07:32:18.720449 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":299,"threads_started":1}
I20260926 07:32:18.723446 22682 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46051
I20260926 07:32:18.723755 22682 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:18.724647 22682 heartbeater.cc:511] Master 127.2.147.254:46051 requested a full tablet report, sending...
I20260926 07:32:18.726831 22496 ts_manager.cc:194] Registered new tserver with Master: edd83b13dc284706bee62411b9c5c689 (127.2.147.193:41795)
I20260926 07:32:18.728403 22496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.193:33019
W20260926 07:32:19.036320 22686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:19.036646 22686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:19.036818 22686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:19.043941 22686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:19.044075 22686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:19.044260 22686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.194
I20260926 07:32:19.053130 22686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.147.194
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.194
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:19.054418 22686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:19.056290 22686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:19.069011 22694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:19.069042 22692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:19.069176 22691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:19.070053 22686 server_base.cc:1034] running on GCE node
I20260926 07:32:19.070645 22686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:19.071586 22686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:19.072810 22686 hybrid_clock.cc:648] HybridClock initialized: now 1790407939072767 us; error 52 us; skew 500 ppm
I20260926 07:32:19.073113 22686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:19.075244 22686 webserver.cc:460] Webserver started at http://127.2.147.194:35081/ using document root <none> and password file <none>
I20260926 07:32:19.075902 22686 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:19.076031 22686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:19.076341 22686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:19.079013 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data/instance:
uuid: "afe202d9c28f47e88e07a23a326c609e"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "19131dc3d39c08da8a5494f9486e978a"
server_key_iv: "72e2565378dd4f866fe96bd9ae5566c3"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.079831 22686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance:
uuid: "afe202d9c28f47e88e07a23a326c609e"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "19131dc3d39c08da8a5494f9486e978a"
server_key_iv: "72e2565378dd4f866fe96bd9ae5566c3"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.084513 22686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:19.087833 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.089067 22686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:19.089238 22686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal
uuid: "afe202d9c28f47e88e07a23a326c609e"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "19131dc3d39c08da8a5494f9486e978a"
server_key_iv: "72e2565378dd4f866fe96bd9ae5566c3"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.089466 22686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:19.130896 22686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:19.131880 22686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:19.132195 22686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:19.133173 22686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:19.134619 22686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:19.134747 22686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.134871 22686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:19.134965 22686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.167354 22686 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.194:44165
I20260926 07:32:19.167376 22812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.194:44165 every 8 connection(s)
I20260926 07:32:19.168687 22686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:19.169843 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22686
I20260926 07:32:19.170022 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:19.170552 2639 external_mini_cluster.cc:1428] Setting key 333937e9f9b622f0a07ebed36244bda0
I20260926 07:32:19.173693 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.147.195:0
--local_ip_for_outbound_sockets=127.2.147.195
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--builtin_ntp_servers=127.2.147.212:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:32:19.184930 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.171535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.194:48657 (local address 127.2.147.254:46051)
0926 07:32:19.171734 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.171740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.173514 (+ 1774us) server_negotiation.cc:407] Connection header received
0926 07:32:19.176413 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.176419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.176496 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.176601 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:19.177969 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.179084 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.181593 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.182228 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.182354 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.184029 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:19.184035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:19.184041 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:19.184063 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:19.184103 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:19.184118 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:19.184208 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.184463 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.184563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:19.186151 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.172212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46051 (local address 127.2.147.194:48657)
0926 07:32:19.173340 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.173388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.176111 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.176847 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.176861 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.176997 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:19.177773 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.177799 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.179296 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.179305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.181430 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.181448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.182481 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.182488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.182817 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.183724 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:19.183727 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:19.183770 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:19.183848 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:19.184225 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:19.184237 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:19.184248 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.184418 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.184839 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":334,"threads_started":1}
I20260926 07:32:19.187724 22813 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46051
I20260926 07:32:19.188028 22813 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:19.188885 22813 heartbeater.cc:511] Master 127.2.147.254:46051 requested a full tablet report, sending...
I20260926 07:32:19.190311 22496 ts_manager.cc:194] Registered new tserver with Master: afe202d9c28f47e88e07a23a326c609e (127.2.147.194:44165)
I20260926 07:32:19.191144 22496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.194:48657
W20260926 07:32:19.504067 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:19.504390 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:19.504570 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:19.512120 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:19.512256 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:19.512434 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.147.195
I20260926 07:32:19.521580 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.147.195
--webserver_port=0
--tserver_master_addrs=127.2.147.254:46051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.147.195
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:19.523077 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:19.524955 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:19.538483 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:19.538451 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:19.538523 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:19.539419 22816 server_base.cc:1034] running on GCE node
I20260926 07:32:19.540071 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:19.541002 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:19.542227 22816 hybrid_clock.cc:648] HybridClock initialized: now 1790407939542176 us; error 63 us; skew 500 ppm
I20260926 07:32:19.542541 22816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 07:32:19.544770 22816 webserver.cc:460] Webserver started at http://127.2.147.195:36201/ using document root <none> and password file <none>
I20260926 07:32:19.545454 22816 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:19.545588 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:19.545912 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:19.548827 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data/instance:
uuid: "f2bb3d703edc406790eec45ce1b518ff"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "1a57d0814e63400ebad1188f6db2d9bc"
server_key_iv: "357406dd224bfe09f087989235072f51"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.549664 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance:
uuid: "f2bb3d703edc406790eec45ce1b518ff"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "1a57d0814e63400ebad1188f6db2d9bc"
server_key_iv: "357406dd224bfe09f087989235072f51"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.554700 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:32:19.558085 22830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.559402 22816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:19.559568 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal
uuid: "f2bb3d703edc406790eec45ce1b518ff"
format_stamp: "Formatted at 2026-09-26 07:32:19 on dist-test-slave-1dwd"
server_key: "1a57d0814e63400ebad1188f6db2d9bc"
server_key_iv: "357406dd224bfe09f087989235072f51"
server_key_version: "encryptionkey@0"
I20260926 07:32:19.559785 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:19.570531 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:19.571419 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:19.571723 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:19.572760 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:19.574208 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:19.574327 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.574465 22816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:19.574563 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:19.607635 22816 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.195:40633
I20260926 07:32:19.607674 22942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.195:40633 every 8 connection(s)
I20260926 07:32:19.608976 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:19.613559 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 22816
I20260926 07:32:19.613720 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:19.614162 2639 external_mini_cluster.cc:1428] Setting key 307dfaab64496a2490fb32a54798f396
I20260926 07:32:19.621888 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.611713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:39439 (local address 127.2.147.254:46051)
0926 07:32:19.611960 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.611966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.613348 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 07:32:19.615598 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.615603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.615696 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.615809 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:19.617132 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.617858 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.619365 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.619784 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.619881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.621226 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:19.621230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:19.621234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:19.621252 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:19.621281 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:19.621292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:19.621358 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.621566 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.621646 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:32:19.623003 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.612307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:46051 (local address 127.2.147.195:39439)
0926 07:32:19.613174 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.613209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.615315 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.616028 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.616039 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.616158 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:19.616980 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.617011 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.617981 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.617985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.619184 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.619196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.619974 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.619978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.620185 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.620934 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:32:19.620937 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:19.620983 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:32:19.621064 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:19.621375 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:19.621391 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:19.621397 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.621545 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.621837 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260926 07:32:19.624356 22943 heartbeater.cc:347] Connected to a master server at 127.2.147.254:46051
I20260926 07:32:19.624619 22943 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:19.625313 22943 heartbeater.cc:511] Master 127.2.147.254:46051 requested a full tablet report, sending...
I20260926 07:32:19.626701 22496 ts_manager.cc:194] Registered new tserver with Master: f2bb3d703edc406790eec45ce1b518ff (127.2.147.195:40633)
I20260926 07:32:19.627655 22496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.147.195:39439
I20260926 07:32:19.629468 2639 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:19.639125 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.633696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.2.147.254:46051)
0926 07:32:19.633938 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.633948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.634316 (+ 368us) server_negotiation.cc:407] Connection header received
0926 07:32:19.634444 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.634447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.634519 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.634597 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:19.635068 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.635799 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.637039 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.637382 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.637486 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.638497 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:19.638501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:19.638505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:19.638521 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:19.638551 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:19.638562 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:19.638629 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.638748 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.638816 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:32:19.642928 22495 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:19.649742 22495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:19.674935 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.661175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:41795 (local address 127.0.0.1:42230)
0926 07:32:19.662756 (+ 1581us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.662791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.663113 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.664555 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.664566 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.664612 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:19.665113 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.665129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.667636 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.667645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.670619 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.670640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.673898 (+ 3258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.673905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.674113 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.674178 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.674321 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.674530 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":1197,"threads_started":1}
I20260926 07:32:19.676070 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.661306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.2.147.193:41795)
0926 07:32:19.662322 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.662331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.662946 (+ 615us) server_negotiation.cc:407] Connection header received
0926 07:32:19.663453 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.663460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.663741 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.663924 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:19.665508 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.667421 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.670869 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.672751 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.672908 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.673212 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.675547 (+ 2335us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.675737 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":701,"threads_started":1}
I20260926 07:32:19.680065 22617 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4e10a38299443c8afc6f2705e50746 (DEFAULT_TABLE table=test-table [id=87c2d32cca6a497797183add0294a643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:19.681627 22617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4e10a38299443c8afc6f2705e50746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:19.682811 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.663163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:44165 (local address 127.0.0.1:52336)
0926 07:32:19.675254 (+ 12091us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.675278 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.675421 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.676005 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.676010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.676034 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:19.676674 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.676688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.678198 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.678203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.680254 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.680268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.681996 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.682004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.682152 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.682211 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.682308 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.682442 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11790,"mutex_wait_us":1,"thread_start_us":11638,"threads_started":1}
I20260926 07:32:19.682879 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.663815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.2.147.194:44165)
0926 07:32:19.674563 (+ 10748us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.674570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.675378 (+ 808us) server_negotiation.cc:407] Connection header received
0926 07:32:19.675595 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.675601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.675835 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.676017 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:19.676875 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.678050 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.680472 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.681804 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.681945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.682221 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.682361 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.682521 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":10458,"threads_started":1}
I20260926 07:32:19.683717 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.661703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.195:40633 (local address 127.0.0.1:44746)
0926 07:32:19.673547 (+ 11844us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.673569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.673700 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.674828 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.674836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.674866 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:19.675292 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.675305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.679293 (+ 3988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.679297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.681069 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.681081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.682963 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.682971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.683138 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.683174 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.683271 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.683428 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11693,"thread_start_us":11930,"threads_started":1}
I20260926 07:32:19.684000 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.662146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.2.147.195:40633)
0926 07:32:19.673785 (+ 11639us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.673797 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.673838 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:19.673957 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.673968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.674262 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.674469 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:19.675522 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.677028 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.681242 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.682442 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.682563 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.682812 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.683458 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.683630 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11430,"thread_start_us":11248,"threads_started":1}
I20260926 07:32:19.686501 22748 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4e10a38299443c8afc6f2705e50746 (DEFAULT_TABLE table=test-table [id=87c2d32cca6a497797183add0294a643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:19.686874 22878 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4e10a38299443c8afc6f2705e50746 (DEFAULT_TABLE table=test-table [id=87c2d32cca6a497797183add0294a643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:19.687834 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4e10a38299443c8afc6f2705e50746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:19.688198 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4e10a38299443c8afc6f2705e50746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:19.698846 22961 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Bootstrap starting.
I20260926 07:32:19.700874 22962 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Bootstrap starting.
I20260926 07:32:19.702515 22963 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Bootstrap starting.
I20260926 07:32:19.703857 22961 tablet_bootstrap.cc:654] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:19.704396 22962 tablet_bootstrap.cc:654] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:19.705137 22961 log.cc:824] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:19.705703 22963 tablet_bootstrap.cc:654] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:19.705899 22962 log.cc:824] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:19.706861 22963 log.cc:824] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:19.728763 22962 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: No bootstrap required, opened a new log
I20260926 07:32:19.728755 22961 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: No bootstrap required, opened a new log
I20260926 07:32:19.729079 22962 ts_tablet_manager.cc:1397] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.025s
I20260926 07:32:19.729094 22961 ts_tablet_manager.cc:1397] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.008s
I20260926 07:32:19.729393 22963 tablet_bootstrap.cc:492] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: No bootstrap required, opened a new log
I20260926 07:32:19.729655 22963 ts_tablet_manager.cc:1397] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.012s
I20260926 07:32:19.731359 22682 heartbeater.cc:503] Master 127.2.147.254:46051 was elected leader, sending a full tablet report...
I20260926 07:32:19.738468 22961 raft_consensus.cc:348] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.738845 22961 raft_consensus.cc:374] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:19.738936 22961 raft_consensus.cc:729] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe202d9c28f47e88e07a23a326c609e, State: Initialized, Role: FOLLOWER
I20260926 07:32:19.739115 22962 raft_consensus.cc:348] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.739560 22962 raft_consensus.cc:374] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:19.739661 22962 raft_consensus.cc:729] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd83b13dc284706bee62411b9c5c689, State: Initialized, Role: FOLLOWER
I20260926 07:32:19.739643 22961 consensus_queue.cc:260] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.740310 22962 consensus_queue.cc:260] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.740813 22813 heartbeater.cc:503] Master 127.2.147.254:46051 was elected leader, sending a full tablet report...
I20260926 07:32:19.741425 22961 ts_tablet_manager.cc:1428] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260926 07:32:19.741634 22963 raft_consensus.cc:348] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.742108 22963 raft_consensus.cc:374] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:19.742213 22963 raft_consensus.cc:729] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2bb3d703edc406790eec45ce1b518ff, State: Initialized, Role: FOLLOWER
I20260926 07:32:19.742754 22963 consensus_queue.cc:260] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.743109 22962 ts_tablet_manager.cc:1428] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260926 07:32:19.744009 22943 heartbeater.cc:503] Master 127.2.147.254:46051 was elected leader, sending a full tablet report...
I20260926 07:32:19.744597 22963 ts_tablet_manager.cc:1428] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 07:32:19.857291 22969 raft_consensus.cc:482] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:19.857697 22969 raft_consensus.cc:504] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.859525 22969 leader_election.cc:288] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd83b13dc284706bee62411b9c5c689 (127.2.147.193:41795), afe202d9c28f47e88e07a23a326c609e (127.2.147.194:44165)
W20260926 07:32:19.860822 22944 tablet.cc:2369] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:19.866086 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.859864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.193:41795 (local address 127.2.147.195:59781)
0926 07:32:19.860055 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.860073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.860196 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.860844 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.860851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.860888 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:19.861141 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.861153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.862549 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.862557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.864350 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.864381 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.865551 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.865555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.865666 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.865690 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.865759 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.865851 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:32:19.866220 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.859859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:59781 (local address 127.2.147.193:41795)
0926 07:32:19.860088 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.860096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.860134 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:32:19.860366 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.860371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.860479 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.860574 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:19.861324 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.862421 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.864532 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.865400 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.865557 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.865632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.865820 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.865961 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:19.867486 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.860465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.147.195:60093 (local address 127.2.147.194:44165)
0926 07:32:19.860642 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.860647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.861429 (+ 782us) server_negotiation.cc:407] Connection header received
0926 07:32:19.861621 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.861625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.861716 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.861806 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:19.862410 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.863445 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.865437 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.866649 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.866804 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.866852 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.867148 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.867307 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:32:19.867241 22637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4e10a38299443c8afc6f2705e50746" candidate_uuid: "f2bb3d703edc406790eec45ce1b518ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd83b13dc284706bee62411b9c5c689" is_pre_election: true
I20260926 07:32:19.867590 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.860325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.194:44165 (local address 127.2.147.195:60093)
0926 07:32:19.861305 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:19.861321 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:19.861458 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:19.861838 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:19.861844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:19.861873 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:19.862203 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.862214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.863668 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.863671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.865252 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:19.865265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.866802 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.866810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:19.866993 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.867025 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:32:19.867125 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:32:19.867281 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":609,"threads_started":1}
I20260926 07:32:19.867871 22637 raft_consensus.cc:2454] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2bb3d703edc406790eec45ce1b518ff in term 0.
I20260926 07:32:19.868454 22768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4e10a38299443c8afc6f2705e50746" candidate_uuid: "f2bb3d703edc406790eec45ce1b518ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe202d9c28f47e88e07a23a326c609e" is_pre_election: true
I20260926 07:32:19.868649 22832 leader_election.cc:302] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edd83b13dc284706bee62411b9c5c689, f2bb3d703edc406790eec45ce1b518ff; no voters:
I20260926 07:32:19.868993 22768 raft_consensus.cc:2454] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2bb3d703edc406790eec45ce1b518ff in term 0.
I20260926 07:32:19.869068 22969 raft_consensus.cc:2781] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:19.869199 22969 raft_consensus.cc:482] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:19.869304 22969 raft_consensus.cc:3037] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:19.871512 22969 raft_consensus.cc:504] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.872062 22969 leader_election.cc:288] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [CANDIDATE]: Term 1 election: Requested vote from peers edd83b13dc284706bee62411b9c5c689 (127.2.147.193:41795), afe202d9c28f47e88e07a23a326c609e (127.2.147.194:44165)
I20260926 07:32:19.872502 22637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4e10a38299443c8afc6f2705e50746" candidate_uuid: "f2bb3d703edc406790eec45ce1b518ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd83b13dc284706bee62411b9c5c689"
I20260926 07:32:19.872538 22768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4e10a38299443c8afc6f2705e50746" candidate_uuid: "f2bb3d703edc406790eec45ce1b518ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe202d9c28f47e88e07a23a326c609e"
I20260926 07:32:19.872694 22637 raft_consensus.cc:3037] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:19.872726 22768 raft_consensus.cc:3037] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:19.874800 22637 raft_consensus.cc:2454] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2bb3d703edc406790eec45ce1b518ff in term 1.
I20260926 07:32:19.874949 22768 raft_consensus.cc:2454] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2bb3d703edc406790eec45ce1b518ff in term 1.
I20260926 07:32:19.875170 22832 leader_election.cc:302] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edd83b13dc284706bee62411b9c5c689, f2bb3d703edc406790eec45ce1b518ff; no voters:
I20260926 07:32:19.875484 22969 raft_consensus.cc:2781] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:19.875972 22969 raft_consensus.cc:686] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 1 LEADER]: Becoming Leader. State: Replica: f2bb3d703edc406790eec45ce1b518ff, State: Running, Role: LEADER
I20260926 07:32:19.876423 22969 consensus_queue.cc:237] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } }
I20260926 07:32:19.880766 22496 catalog_manager.cc:5495] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff reported cstate change: term changed from 0 to 1, leader changed from <none> to f2bb3d703edc406790eec45ce1b518ff (127.2.147.195). New cstate: current_term: 1 leader_uuid: "f2bb3d703edc406790eec45ce1b518ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2bb3d703edc406790eec45ce1b518ff" member_type: VOTER last_known_addr { host: "127.2.147.195" port: 40633 } health_report { overall_health: HEALTHY } } }
I20260926 07:32:19.919791 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.914743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.2.147.195:40633)
0926 07:32:19.914932 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.914937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.915066 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:32:19.915311 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.915315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.915393 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.915529 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:19.915831 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.916554 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.917837 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.918216 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.918302 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.918757 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:19.919326 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:19.919444 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.919532 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.919596 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:32:19.920615 22814 tablet.cc:2369] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:19.928013 22637 raft_consensus.cc:1261] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 1 FOLLOWER]: Refusing update from remote peer f2bb3d703edc406790eec45ce1b518ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:19.928032 22768 raft_consensus.cc:1261] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 1 FOLLOWER]: Refusing update from remote peer f2bb3d703edc406790eec45ce1b518ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:19.928932 22971 consensus_queue.cc:1035] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe202d9c28f47e88e07a23a326c609e" member_type: VOTER last_known_addr { host: "127.2.147.194" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:19.929311 22969 consensus_queue.cc:1035] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd83b13dc284706bee62411b9c5c689" member_type: VOTER last_known_addr { host: "127.2.147.193" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:19.940721 22974 mvcc.cc:205] Tried to move back new op lower bound from 7333510921934815232 to 7333510921737490432. Current Snapshot: MvccSnapshot[applied={T|T < 7333510921934815232}]
I20260926 07:32:19.941052 22975 mvcc.cc:205] Tried to move back new op lower bound from 7333510921934815232 to 7333510921737490432. Current Snapshot: MvccSnapshot[applied={T|T < 7333510921934815232}]
I20260926 07:32:19.953049 22494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:19.953313 22494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:19.956861 22494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26f74f9d099340db9c95a7554b3f2a30: Sending DeleteTablet for 3 replicas of tablet 8d4e10a38299443c8afc6f2705e50746
I20260926 07:32:19.957916 22878 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4e10a38299443c8afc6f2705e50746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:19 UTC) from {username='slave'} at 127.0.0.1:44746
I20260926 07:32:19.958132 22748 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4e10a38299443c8afc6f2705e50746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:19 UTC) from {username='slave'} at 127.0.0.1:52336
I20260926 07:32:19.958154 22617 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4e10a38299443c8afc6f2705e50746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:19 UTC) from {username='slave'} at 127.0.0.1:42230
I20260926 07:32:19.959197 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22555
I20260926 07:32:19.959367 22984 tablet_replica.cc:330] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: stopping tablet replica
I20260926 07:32:19.959827 22984 raft_consensus.cc:2229] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:19.960809 22985 tablet_replica.cc:330] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: stopping tablet replica
I20260926 07:32:19.961318 22985 raft_consensus.cc:2229] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:19.962637 22984 raft_consensus.cc:2258] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:19.963290 22985 raft_consensus.cc:2258] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:19.966097 22984 ts_tablet_manager.cc:1905] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:19.972703 22985 ts_tablet_manager.cc:1905] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:19.973485 22983 tablet_replica.cc:330] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: stopping tablet replica
I20260926 07:32:19.973867 22983 raft_consensus.cc:2229] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:19.974472 22983 raft_consensus.cc:2258] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:19.976445 22983 ts_tablet_manager.cc:1905] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:19.980434 22984 ts_tablet_manager.cc:1918] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:19.980592 22984 log.cc:1196] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-1/wal/wals/8d4e10a38299443c8afc6f2705e50746
I20260926 07:32:19.981144 22984 ts_tablet_manager.cc:1939] T 8d4e10a38299443c8afc6f2705e50746 P afe202d9c28f47e88e07a23a326c609e: Deleting consensus metadata
I20260926 07:32:19.982995 22481 catalog_manager.cc:4841] TS afe202d9c28f47e88e07a23a326c609e (127.2.147.194:44165): tablet 8d4e10a38299443c8afc6f2705e50746 (table test-table [id=87c2d32cca6a497797183add0294a643]) successfully deleted
I20260926 07:32:19.983630 22983 ts_tablet_manager.cc:1918] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:19.983668 22985 ts_tablet_manager.cc:1918] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:19.983776 22983 log.cc:1196] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-2/wal/wals/8d4e10a38299443c8afc6f2705e50746
I20260926 07:32:19.983831 22985 log.cc:1196] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Deleting WAL directory at /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407623164223-2639-0/minicluster-data/ts-0/wal/wals/8d4e10a38299443c8afc6f2705e50746
I20260926 07:32:19.984298 22983 ts_tablet_manager.cc:1939] T 8d4e10a38299443c8afc6f2705e50746 P f2bb3d703edc406790eec45ce1b518ff: Deleting consensus metadata
I20260926 07:32:19.984338 22985 ts_tablet_manager.cc:1939] T 8d4e10a38299443c8afc6f2705e50746 P edd83b13dc284706bee62411b9c5c689: Deleting consensus metadata
I20260926 07:32:19.985874 22480 catalog_manager.cc:4841] TS f2bb3d703edc406790eec45ce1b518ff (127.2.147.195:40633): tablet 8d4e10a38299443c8afc6f2705e50746 (table test-table [id=87c2d32cca6a497797183add0294a643]) successfully deleted
I20260926 07:32:19.986012 22481 catalog_manager.cc:4841] TS edd83b13dc284706bee62411b9c5c689 (127.2.147.193:41795): tablet 8d4e10a38299443c8afc6f2705e50746 (table test-table [id=87c2d32cca6a497797183add0294a643]) successfully deleted
I20260926 07:32:19.988350 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:19.961333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.2.147.193:41795)
0926 07:32:19.961545 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:19.961554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:19.977065 (+ 15511us) server_negotiation.cc:407] Connection header received
0926 07:32:19.980692 (+ 3627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:19.980709 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:19.980820 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:19.980898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:19.981674 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.982784 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.984870 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:19.985461 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:19.985577 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:19.987040 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:19.987047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:19.987059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:19.987646 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:19.987688 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:19.987706 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:19.987831 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:19.987931 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:32:19.988032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:32:19.989027 22677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:20.076368 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22555
I20260926 07:32:20.093695 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22686
I20260926 07:32:20.100654 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:20.094497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.2.147.194:44165)
0926 07:32:20.094922 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:20.094927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:20.094942 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:32:20.094995 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:20.094999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:20.095093 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:20.095221 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:20.095641 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:20.096351 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:20.097621 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:20.098024 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:20.098162 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:20.099324 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:20.099332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:20.099346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:20.099914 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:20.099959 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:20.099978 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:20.100118 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:20.100242 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:32:20.100344 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260926 07:32:20.101325 22808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:20.197907 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22686
I20260926 07:32:20.215698 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22816
I20260926 07:32:20.223680 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:20.216519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.2.147.195:40633)
0926 07:32:20.216704 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:20.216708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:20.216773 (+ 65us) server_negotiation.cc:407] Connection header received
0926 07:32:20.216996 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:20.217000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:20.217100 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:20.217198 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:20.217690 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:20.218488 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:20.220395 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:20.220824 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:20.220917 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:20.222128 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:20.222137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:20.222155 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:20.222752 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:20.222802 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:20.222822 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:20.222909 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:20.223091 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:32:20.223172 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:20.224498 22938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:20.345813 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22816
I20260926 07:32:20.365368 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 22465
I20260926 07:32:20.366451 22526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:20.479269 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 22465
2026-09-26T07:32:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2815 ms)
[----------] 6 tests from EncryptionPolicyTest (21964 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:20 dist-test-slave-1dwd krb5kdc[22993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:20 dist-test-slave-1dwd krb5kdc[22993](info): set up 2 sockets
Sep 26 07:32:20 dist-test-slave-1dwd krb5kdc[22993](info): commencing operation
krb5kdc: starting...
W20260926 07:32:21.595777 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:21 dist-test-slave-1dwd krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:21.850409 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:22.178135 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:22.178411 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:22.178503 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:22.185547 23006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:22.185658 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:22.185719 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:22.185781 23006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:22.185825 23006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:22.195705 23006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:22.196911 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:22.198170 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:22.209661 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:22.209734 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:22.209705 23014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:22.210268 23006 server_base.cc:1034] running on GCE node
I20260926 07:32:22.211100 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:22.212548 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:22.213747 23006 hybrid_clock.cc:648] HybridClock initialized: now 1790407942213710 us; error 55 us; skew 500 ppm
I20260926 07:32:22.214097 23006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:22 dist-test-slave-1dwd krb5kdc[22993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:22.222615 23006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:22.224181 23006 webserver.cc:460] Webserver started at http://127.0.0.1:41273/ using document root <none> and password file <none>
I20260926 07:32:22.224850 23006 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:22.224953 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:22.225219 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:22.228309 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "94d289dffeb14a8d95a60bff208303aa"
format_stamp: "Formatted at 2026-09-26 07:32:22 on dist-test-slave-1dwd"
server_key: "1be2b0e6da875361132ded567981e603"
server_key_iv: "0cd74f2d5f0ff9aeeb8e11eedd8f8499"
server_key_version: "encryptionkey@0"
I20260926 07:32:22.229128 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "94d289dffeb14a8d95a60bff208303aa"
format_stamp: "Formatted at 2026-09-26 07:32:22 on dist-test-slave-1dwd"
server_key: "1be2b0e6da875361132ded567981e603"
server_key_iv: "0cd74f2d5f0ff9aeeb8e11eedd8f8499"
server_key_version: "encryptionkey@0"
I20260926 07:32:22.234475 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:32:22.238257 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:22.239743 23006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:32:22.239940 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "94d289dffeb14a8d95a60bff208303aa"
format_stamp: "Formatted at 2026-09-26 07:32:22 on dist-test-slave-1dwd"
server_key: "1be2b0e6da875361132ded567981e603"
server_key_iv: "0cd74f2d5f0ff9aeeb8e11eedd8f8499"
server_key_version: "encryptionkey@0"
I20260926 07:32:22.240147 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:22.253151 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:22.257181 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:22.257483 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:22.275856 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42437 every 8 connection(s)
I20260926 07:32:22.275861 23006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42437
I20260926 07:32:22.277338 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:22.281085 23073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:22.284993 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23006
I20260926 07:32:22.285212 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:22.285996 2639 external_mini_cluster.cc:1428] Setting key 31c89accf0ad794b3907c77c53abcc29
I20260926 07:32:22.294657 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: Bootstrap starting.
I20260926 07:32:22.298516 23073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:22.299902 23073 log.cc:824] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:22 dist-test-slave-1dwd krb5kdc[22993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:32:22.303290 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: No bootstrap required, opened a new log
I20260926 07:32:22.310659 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:22.288040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39655 (local address 127.0.0.1:42437)
0926 07:32:22.288862 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:22.288874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:22.288915 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:22.290517 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:22.290539 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:22.290954 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:22.291340 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:22.293007 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:22.294294 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:22.296079 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:22.296774 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:22.296907 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:22.301987 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:22.302000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:22.302017 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:22.302058 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:22.306076 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:22.306908 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:22.306921 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:22.306929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:22.307134 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:22.307606 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:22.307611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:22.307615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:22.308272 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:22.308611 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:22.308942 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:32:22.309292 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":412,"threads_started":1}
I20260926 07:32:22.313402 23073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260926 07:32:22.313853 23073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:22.313987 23073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d289dffeb14a8d95a60bff208303aa, State: Initialized, Role: FOLLOWER
I20260926 07:32:22.314751 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:22.314677 23073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260926 07:32:22.314961 23073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:22.315085 23073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:22.315266 23073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:22.318495 23073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260926 07:32:22.319180 23073 leader_election.cc:302] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d289dffeb14a8d95a60bff208303aa; no voters:
I20260926 07:32:22.320012 23073 leader_election.cc:288] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:22.320086 23082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:22.322190 23073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:22.328419 23082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [term 1 LEADER]: Becoming Leader. State: Replica: 94d289dffeb14a8d95a60bff208303aa, State: Running, Role: LEADER
I20260926 07:32:22.329210 23082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } }
I20260926 07:32:22.332217 23083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d289dffeb14a8d95a60bff208303aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } } }
I20260926 07:32:22.332626 23083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:22.333154 23084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d289dffeb14a8d95a60bff208303aa. Latest consensus state: current_term: 1 leader_uuid: "94d289dffeb14a8d95a60bff208303aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d289dffeb14a8d95a60bff208303aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42437 } } }
I20260926 07:32:22.333437 23084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa [sys.catalog]: This master's current role is: LEADER
W20260926 07:32:22.339049 23098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:22.339226 23098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:22.339444 23095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:22.341874 23095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:22.345947 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:22.321342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50925 (local address 127.0.0.1:42437)
0926 07:32:22.321526 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:22.321534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:22.327949 (+ 6415us) server_negotiation.cc:407] Connection header received
0926 07:32:22.328222 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:22.328230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:22.328340 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:22.328456 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:22.330635 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:22.332367 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:22.334451 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:22.335090 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:22.335278 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:22.338264 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:22.338270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:22.338275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:22.338311 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:22.342364 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:22.343328 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:22.343334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:22.343337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:22.343496 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:22.344095 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:22.344105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:22.344108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:22.344590 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:22.344855 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:22.345377 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 07:32:22.345559 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:32:22.350894 23095 catalog_manager.cc:1347] Generated new cluster ID: 29c3b4d312744906a92ffd94a2637afe
I20260926 07:32:22.351060 23095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:22.374256 23095 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:22.376084 23095 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:22.383940 23095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d289dffeb14a8d95a60bff208303aa: Generated new TSK 0
I20260926 07:32:22.384794 23095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:22.399075 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23006
I20260926 07:32:22.400285 23068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:22.493417 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23006
W20260926 07:32:22.510665 22991 connection.cc:535] client connection to 127.0.0.1:42437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42437): Connection reset by peer (error 104)
2026-09-26T07:32:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:22 dist-test-slave-1dwd krb5kdc[23105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:22 dist-test-slave-1dwd krb5kdc[23105](info): set up 2 sockets
Sep 26 07:32:22 dist-test-slave-1dwd krb5kdc[23105](info): commencing operation
krb5kdc: starting...
W20260926 07:32:23.611938 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:23 dist-test-slave-1dwd krb5kdc[23105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:23.852571 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:24.176414 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:24.176748 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:24.176903 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:24.183954 23118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:24.184110 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:24.184194 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:24.184283 23118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:24.184376 23118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:24.193988 23118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:24.195345 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:24.196709 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:24.208529 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:24.208469 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:24.208563 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:24.209309 23118 server_base.cc:1034] running on GCE node
I20260926 07:32:24.210090 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:24.211540 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:24.212761 23118 hybrid_clock.cc:648] HybridClock initialized: now 1790407944212706 us; error 50 us; skew 500 ppm
I20260926 07:32:24.213111 23118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:24 dist-test-slave-1dwd krb5kdc[23105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:24.216135 23118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:24.217521 23118 webserver.cc:460] Webserver started at http://127.0.0.1:38413/ using document root <none> and password file <none>
I20260926 07:32:24.218156 23118 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:24.218252 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:24.218492 23118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:24.221405 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "7a80499503bb4d18aaa86ffaa84b9422"
format_stamp: "Formatted at 2026-09-26 07:32:24 on dist-test-slave-1dwd"
server_key: "f71c65b67959a446e204ddd67822e8dc"
server_key_iv: "ab84ae51979cd28bed9720a0dcd1baa6"
server_key_version: "encryptionkey@0"
I20260926 07:32:24.222107 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "7a80499503bb4d18aaa86ffaa84b9422"
format_stamp: "Formatted at 2026-09-26 07:32:24 on dist-test-slave-1dwd"
server_key: "f71c65b67959a446e204ddd67822e8dc"
server_key_iv: "ab84ae51979cd28bed9720a0dcd1baa6"
server_key_version: "encryptionkey@0"
I20260926 07:32:24.227030 23118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:32:24.230471 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:24.231956 23118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:24.232154 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "7a80499503bb4d18aaa86ffaa84b9422"
format_stamp: "Formatted at 2026-09-26 07:32:24 on dist-test-slave-1dwd"
server_key: "f71c65b67959a446e204ddd67822e8dc"
server_key_iv: "ab84ae51979cd28bed9720a0dcd1baa6"
server_key_version: "encryptionkey@0"
I20260926 07:32:24.232353 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:24.253396 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:24.257300 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:24.257581 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:24.275655 23118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34187
I20260926 07:32:24.275748 23184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34187 every 8 connection(s)
I20260926 07:32:24.277017 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:24.280709 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:24.286124 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23118
I20260926 07:32:24.286302 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:24.286751 2639 external_mini_cluster.cc:1428] Setting key dd364f9c53738e6cc82ef7fc5208c2f6
I20260926 07:32:24.291923 23185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422: Bootstrap starting.
I20260926 07:32:24.295475 23185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:24.296626 23185 log.cc:824] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:24.299804 23185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422: No bootstrap required, opened a new log
I20260926 07:32:24.300172 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:24.288547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54309 (local address 127.0.0.1:34187)
0926 07:32:24.289373 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:24.289387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:24.289431 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:32:24.291052 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:24.291092 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:24.291549 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:24.291869 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:24.292443 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:24.293728 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:24.295607 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:24.296284 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:24.296406 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:24.297744 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:24.297760 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:24.297777 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:24.298410 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:24.298482 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:24.298506 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:24.298624 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:24.298898 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 07:32:24.299102 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":363,"threads_started":1}
I20260926 07:32:24.303879 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:24.309250 23185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260926 07:32:24.309614 23185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:24.309751 23185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a80499503bb4d18aaa86ffaa84b9422, State: Initialized, Role: FOLLOWER
I20260926 07:32:24.310336 23185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260926 07:32:24.310657 23185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:24.310797 23185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:24.310981 23185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:24.313706 23185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260926 07:32:24.314244 23185 leader_election.cc:302] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a80499503bb4d18aaa86ffaa84b9422; no voters:
I20260926 07:32:24.314859 23185 leader_election.cc:288] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:24.314993 23195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:24.315932 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:24.308948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45531 (local address 127.0.0.1:34187)
0926 07:32:24.309130 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:24.309136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:24.309548 (+ 412us) server_negotiation.cc:407] Connection header received
0926 07:32:24.309733 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:24.309744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:24.309823 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:24.309931 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:24.310454 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:24.311614 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:24.312833 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:24.313525 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:24.313652 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:24.314859 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:24.314869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:24.314875 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:24.314899 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:24.314943 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:24.314964 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:24.315082 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:24.315412 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 07:32:24.315480 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:32:24.317037 23185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:24.318149 23195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [term 1 LEADER]: Becoming Leader. State: Replica: 7a80499503bb4d18aaa86ffaa84b9422, State: Running, Role: LEADER
I20260926 07:32:24.318817 23195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } }
I20260926 07:32:24.322247 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a80499503bb4d18aaa86ffaa84b9422. Latest consensus state: current_term: 1 leader_uuid: "7a80499503bb4d18aaa86ffaa84b9422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } } }
I20260926 07:32:24.322760 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:24.323633 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a80499503bb4d18aaa86ffaa84b9422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a80499503bb4d18aaa86ffaa84b9422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34187 } } }
I20260926 07:32:24.323953 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:24.324246 23206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:24.329363 23206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:24.338737 23206 catalog_manager.cc:1347] Generated new cluster ID: 5aceab6f54ab423fb5f86e2f3ebcaa2f
I20260926 07:32:24.338845 23206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:24.361622 23206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:24.363461 23206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:24.374833 23206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a80499503bb4d18aaa86ffaa84b9422: Generated new TSK 0
I20260926 07:32:24.375908 23206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:24.390553 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23118
W20260926 07:32:24.391692 23180 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54309
W20260926 07:32:24.392602 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23118: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:24.392791 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23118
W20260926 07:32:24.408008 23103 connection.cc:535] client connection to 127.0.0.1:34187 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34187): Connection reset by peer (error 104)
2026-09-26T07:32:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:24 dist-test-slave-1dwd krb5kdc[23216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:24 dist-test-slave-1dwd krb5kdc[23216](info): set up 2 sockets
Sep 26 07:32:24 dist-test-slave-1dwd krb5kdc[23216](info): commencing operation
krb5kdc: starting...
W20260926 07:32:25.503572 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:25 dist-test-slave-1dwd krb5kdc[23216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:25.750675 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:26.076577 23229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:26.076853 23229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:26.076973 23229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:26.084036 23229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:26.084165 23229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:26.084271 23229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:26.084373 23229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:26.084432 23229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:26.094616 23229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:26.095991 23229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:26.097296 23229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:26.109163 23237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:26.109210 23235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:26.109256 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:26.110142 23229 server_base.cc:1034] running on GCE node
I20260926 07:32:26.110920 23229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:26.112339 23229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:26.113569 23229 hybrid_clock.cc:648] HybridClock initialized: now 1790407946113526 us; error 58 us; skew 500 ppm
I20260926 07:32:26.113895 23229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:26 dist-test-slave-1dwd krb5kdc[23216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:26.116928 23229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:26.118304 23229 webserver.cc:460] Webserver started at http://127.0.0.1:35771/ using document root <none> and password file <none>
I20260926 07:32:26.118924 23229 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:26.119035 23229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:26.119370 23229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:26.122318 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "dbea85c239044db89a346b8b36b9f2f6"
format_stamp: "Formatted at 2026-09-26 07:32:26 on dist-test-slave-1dwd"
server_key: "f03ea40290d950eb534888cb27b1076d"
server_key_iv: "2026a1747ccd17356913b19a19da8dba"
server_key_version: "encryptionkey@0"
I20260926 07:32:26.123080 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "dbea85c239044db89a346b8b36b9f2f6"
format_stamp: "Formatted at 2026-09-26 07:32:26 on dist-test-slave-1dwd"
server_key: "f03ea40290d950eb534888cb27b1076d"
server_key_iv: "2026a1747ccd17356913b19a19da8dba"
server_key_version: "encryptionkey@0"
I20260926 07:32:26.128679 23229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:32:26.132099 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:26.133399 23229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:26.133538 23229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "dbea85c239044db89a346b8b36b9f2f6"
format_stamp: "Formatted at 2026-09-26 07:32:26 on dist-test-slave-1dwd"
server_key: "f03ea40290d950eb534888cb27b1076d"
server_key_iv: "2026a1747ccd17356913b19a19da8dba"
server_key_version: "encryptionkey@0"
I20260926 07:32:26.133709 23229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:26.139961 23229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:26.143747 23229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:26.144043 23229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:26.162006 23229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42899
I20260926 07:32:26.162053 23295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42899 every 8 connection(s)
I20260926 07:32:26.163435 23229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:26.164661 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23229
I20260926 07:32:26.164851 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:26.165364 2639 external_mini_cluster.cc:1428] Setting key da148e28baf37ac17962a2e10d9b2d47
I20260926 07:32:26.167922 23296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:26.172961 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:26.167112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42955 (local address 127.0.0.1:42899)
0926 07:32:26.168032 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:26.168047 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:26.168100 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:32:26.169669 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:26.169689 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:26.170005 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:26.170323 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:26.171377 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:26.171383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:26.171394 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:26.171410 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:26.171486 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:26.171507 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:26.171671 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:26.171852 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:32:26.172056 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":365,"threads_started":1}
I20260926 07:32:26.176548 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:26.180418 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6: Bootstrap starting.
I20260926 07:32:26.184494 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:26.181856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56993 (local address 127.0.0.1:42899)
0926 07:32:26.182008 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:26.182012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:26.182359 (+ 347us) server_negotiation.cc:407] Connection header received
0926 07:32:26.182543 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:26.182547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:26.182592 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:26.182666 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:26.183720 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:26.183725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:26.183728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:26.183742 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:26.183804 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:26.183819 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:26.183911 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:26.184102 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:32:26.184151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:32:26.184520 23296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:26.185669 23296 log.cc:824] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:26.188293 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6: No bootstrap required, opened a new log
I20260926 07:32:26.195525 23296 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260926 07:32:26.195940 23296 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:26.196054 23296 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbea85c239044db89a346b8b36b9f2f6, State: Initialized, Role: FOLLOWER
I20260926 07:32:26.196633 23296 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260926 07:32:26.196882 23296 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:26.197031 23296 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:26.197208 23296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:26.199445 23296 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260926 07:32:26.199940 23296 leader_election.cc:302] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbea85c239044db89a346b8b36b9f2f6; no voters:
I20260926 07:32:26.200531 23296 leader_election.cc:288] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:26.200675 23306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:26.201725 23306 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [term 1 LEADER]: Becoming Leader. State: Replica: dbea85c239044db89a346b8b36b9f2f6, State: Running, Role: LEADER
I20260926 07:32:26.202296 23306 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260926 07:32:26.202845 23296 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:26.205060 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbea85c239044db89a346b8b36b9f2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260926 07:32:26.205477 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:26.206007 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbea85c239044db89a346b8b36b9f2f6. Latest consensus state: current_term: 1 leader_uuid: "dbea85c239044db89a346b8b36b9f2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea85c239044db89a346b8b36b9f2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260926 07:32:26.206339 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:26.206604 23313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:26.213428 23313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:26.225455 23313 catalog_manager.cc:1347] Generated new cluster ID: b2da976e11e74aa493469315226a565c
I20260926 07:32:26.225592 23313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:26.243060 23313 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:26.243533 23313 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:26.266944 23313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbea85c239044db89a346b8b36b9f2f6: Generated new TSK 0
I20260926 07:32:26.268023 23313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:26.281489 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23229
W20260926 07:32:26.282655 23291 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42955
W20260926 07:32:26.283362 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:26.283511 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23229
2026-09-26T07:32:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:26 dist-test-slave-1dwd krb5kdc[23328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:26 dist-test-slave-1dwd krb5kdc[23328](info): set up 2 sockets
Sep 26 07:32:26 dist-test-slave-1dwd krb5kdc[23328](info): commencing operation
krb5kdc: starting...
W20260926 07:32:27.383662 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:27 dist-test-slave-1dwd krb5kdc[23328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:27.636729 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:27.962779 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:27.963129 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:27.963310 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:27.970711 23341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:27.970873 23341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:27.970988 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:27.971091 23341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:27.971174 23341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:27.981117 23341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:27.982419 23341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:27.983742 23341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:27.995253 23347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:27.995277 23349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:27.995249 23346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:27.996337 23341 server_base.cc:1034] running on GCE node
I20260926 07:32:27.997061 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:27.998389 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:27.999619 23341 hybrid_clock.cc:648] HybridClock initialized: now 1790407947999575 us; error 55 us; skew 500 ppm
I20260926 07:32:27.999959 23341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:28 dist-test-slave-1dwd krb5kdc[23328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:28.008678 23341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:28.010133 23341 webserver.cc:460] Webserver started at http://127.0.0.1:39311/ using document root <none> and password file <none>
I20260926 07:32:28.010777 23341 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:28.010870 23341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:28.011116 23341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:28.014035 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "4975fbff852c49beaf472b0814723e19"
format_stamp: "Formatted at 2026-09-26 07:32:28 on dist-test-slave-1dwd"
server_key: "cfcc9e2dc7ca1c5920d294f565488ed5"
server_key_iv: "77cb8c1ce04c8b8002dc38bf8b30280b"
server_key_version: "encryptionkey@0"
I20260926 07:32:28.014714 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "4975fbff852c49beaf472b0814723e19"
format_stamp: "Formatted at 2026-09-26 07:32:28 on dist-test-slave-1dwd"
server_key: "cfcc9e2dc7ca1c5920d294f565488ed5"
server_key_iv: "77cb8c1ce04c8b8002dc38bf8b30280b"
server_key_version: "encryptionkey@0"
I20260926 07:32:28.019740 23341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 07:32:28.023288 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:28.024675 23341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:28.024876 23341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "4975fbff852c49beaf472b0814723e19"
format_stamp: "Formatted at 2026-09-26 07:32:28 on dist-test-slave-1dwd"
server_key: "cfcc9e2dc7ca1c5920d294f565488ed5"
server_key_iv: "77cb8c1ce04c8b8002dc38bf8b30280b"
server_key_version: "encryptionkey@0"
I20260926 07:32:28.025074 23341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:28.036599 23341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:28.040999 23341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:28.041298 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:28.059641 23341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46441
I20260926 07:32:28.059674 23407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46441 every 8 connection(s)
I20260926 07:32:28.061013 23341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:28.064605 23408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:28.070313 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23341
I20260926 07:32:28.070485 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:28.070915 2639 external_mini_cluster.cc:1428] Setting key e5e6b407ede036730af8bedf4f62a4ff
I20260926 07:32:28.075309 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19: Bootstrap starting.
I20260926 07:32:28.078938 23408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:28.080229 23408 log.cc:824] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:28.083410 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19: No bootstrap required, opened a new log
Sep 26 07:32:28 dist-test-slave-1dwd krb5kdc[23328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:32:28.091888 23408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260926 07:32:28.092267 23408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:28.092397 23408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4975fbff852c49beaf472b0814723e19, State: Initialized, Role: FOLLOWER
I20260926 07:32:28.093142 23408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260926 07:32:28.093441 23408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:28.093598 23408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:28.093776 23408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:28.094069 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:28.072793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34661 (local address 127.0.0.1:46441)
0926 07:32:28.073622 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:28.073634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:28.073672 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:32:28.075341 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:28.075363 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:28.075789 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:28.076096 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:28.077936 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:28.079274 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:28.081107 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:28.081964 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:28.082088 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:28.086693 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:28.086703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:28.086717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:28.086750 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:28.089850 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:28.090540 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:28.090547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:28.090556 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:28.090726 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:28.091115 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:28.091120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:28.091125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:28.091692 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:28.092018 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:28.092427 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 07:32:28.092841 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260926 07:32:28.096246 23408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260926 07:32:28.096800 23408 leader_election.cc:302] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4975fbff852c49beaf472b0814723e19; no voters:
I20260926 07:32:28.097426 23408 leader_election.cc:288] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:28.097580 23413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:28.097899 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:28.099388 23408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:28.108199 23413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [term 1 LEADER]: Becoming Leader. State: Replica: 4975fbff852c49beaf472b0814723e19, State: Running, Role: LEADER
I20260926 07:32:28.108911 23413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } }
I20260926 07:32:28.113153 23415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4975fbff852c49beaf472b0814723e19. Latest consensus state: current_term: 1 leader_uuid: "4975fbff852c49beaf472b0814723e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260926 07:32:28.113567 23415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:28.114138 23414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4975fbff852c49beaf472b0814723e19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975fbff852c49beaf472b0814723e19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46441 } } }
I20260926 07:32:28.114413 23414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:28.116133 23432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:28.121778 23432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:28.128783 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:28.109832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36601 (local address 127.0.0.1:46441)
0926 07:32:28.110068 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:28.110078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:28.111561 (+ 1483us) server_negotiation.cc:407] Connection header received
0926 07:32:28.111797 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:28.111805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:28.111926 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:28.112048 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:28.114289 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:28.115609 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:28.117435 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:28.118205 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:28.118337 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:28.121702 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:28.121709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:28.121713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:28.121761 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:28.125822 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:28.126569 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:28.126574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:28.126578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:28.126718 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:28.127111 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:28.127117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:28.127120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:28.127543 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:28.127797 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:28.128146 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:32:28.128403 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:32:28.132133 23432 catalog_manager.cc:1347] Generated new cluster ID: c319e7da1d4b499db200255194afebdc
I20260926 07:32:28.132258 23432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:28.145864 23432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:28.147754 23432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:28.160784 23432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4975fbff852c49beaf472b0814723e19: Generated new TSK 0
I20260926 07:32:28.161828 23432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:28.168982 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23341
I20260926 07:32:28.170313 23403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:28.261354 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23341
2026-09-26T07:32:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:28 dist-test-slave-1dwd krb5kdc[23440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:28 dist-test-slave-1dwd krb5kdc[23440](info): set up 2 sockets
Sep 26 07:32:28 dist-test-slave-1dwd krb5kdc[23440](info): commencing operation
krb5kdc: starting...
W20260926 07:32:29.371620 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:29 dist-test-slave-1dwd krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:29.626230 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:29.951293 23453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:29.951546 23453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:29.951725 23453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:29.958572 23453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:29.958688 23453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:29.958794 23453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:29.958886 23453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:29.958961 23453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:29.968400 23453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:29.969596 23453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:29.970824 23453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:29.982012 23459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:29.982038 23461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:29.982071 23458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:29.982686 23453 server_base.cc:1034] running on GCE node
I20260926 07:32:29.983549 23453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:29.984830 23453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:29.986028 23453 hybrid_clock.cc:648] HybridClock initialized: now 1790407949985999 us; error 44 us; skew 500 ppm
I20260926 07:32:29.986359 23453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:29 dist-test-slave-1dwd krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:29.989207 23453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:29.990497 23453 webserver.cc:460] Webserver started at http://127.0.0.1:44201/ using document root <none> and password file <none>
I20260926 07:32:29.991103 23453 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:29.991250 23453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:29.991549 23453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:29.994359 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "8b3250612e064d93a3a50a95b95ea502"
format_stamp: "Formatted at 2026-09-26 07:32:29 on dist-test-slave-1dwd"
server_key: "40957fc1e279e12953f2f60e4b384a1c"
server_key_iv: "09eeb1c3c3f43a9aa365b373e2152e2d"
server_key_version: "encryptionkey@0"
I20260926 07:32:29.995123 23453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "8b3250612e064d93a3a50a95b95ea502"
format_stamp: "Formatted at 2026-09-26 07:32:29 on dist-test-slave-1dwd"
server_key: "40957fc1e279e12953f2f60e4b384a1c"
server_key_iv: "09eeb1c3c3f43a9aa365b373e2152e2d"
server_key_version: "encryptionkey@0"
I20260926 07:32:30.000252 23453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:32:30.003744 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:30.005060 23453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:32:30.005228 23453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "8b3250612e064d93a3a50a95b95ea502"
format_stamp: "Formatted at 2026-09-26 07:32:29 on dist-test-slave-1dwd"
server_key: "40957fc1e279e12953f2f60e4b384a1c"
server_key_iv: "09eeb1c3c3f43a9aa365b373e2152e2d"
server_key_version: "encryptionkey@0"
I20260926 07:32:30.005410 23453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:30.028275 23453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:30.032060 23453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:30.032344 23453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:30.050146 23453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34455
I20260926 07:32:30.050176 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34455 every 8 connection(s)
I20260926 07:32:30.051522 23453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:30.055083 23520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:30.060050 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23453
I20260926 07:32:30.060266 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:30.060776 2639 external_mini_cluster.cc:1428] Setting key 6abf55ebc853cb0379d8dc2461126036
I20260926 07:32:30.066094 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502: Bootstrap starting.
I20260926 07:32:30.069670 23520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:30.070853 23520 log.cc:824] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:30.074194 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:30.062534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46167 (local address 127.0.0.1:34455)
0926 07:32:30.063391 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:30.063403 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:30.063447 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:32:30.065130 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:30.065153 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:30.065542 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:30.065851 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:30.066419 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:30.067792 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:30.069617 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:30.070348 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:30.070499 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:30.071728 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:30.071752 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:30.071774 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:30.072582 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:30.072664 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:30.072689 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:30.072780 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:30.072995 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:32:30.073264 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":393,"threads_started":1}
I20260926 07:32:30.074222 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502: No bootstrap required, opened a new log
I20260926 07:32:30.077702 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:30.082099 23520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } }
I20260926 07:32:30.082564 23520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:30.082653 23520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3250612e064d93a3a50a95b95ea502, State: Initialized, Role: FOLLOWER
I20260926 07:32:30.083314 23520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } }
I20260926 07:32:30.083561 23520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:30.083709 23520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:30.083887 23520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:30.085983 23520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } }
I20260926 07:32:30.086496 23520 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3250612e064d93a3a50a95b95ea502; no voters:
I20260926 07:32:30.087248 23520 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:30.087332 23530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:30.087939 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:30.081807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55711 (local address 127.0.0.1:34455)
0926 07:32:30.082014 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:30.082021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:30.082353 (+ 332us) server_negotiation.cc:407] Connection header received
0926 07:32:30.082563 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:30.082573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:30.082654 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:30.082766 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:30.083275 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:30.084161 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:30.085417 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:30.085868 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:30.086001 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:30.087050 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:30.087056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:30.087059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:30.087075 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:30.087110 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:30.087127 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:30.087262 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:30.087405 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 07:32:30.087525 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:30.088558 23530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3250612e064d93a3a50a95b95ea502, State: Running, Role: LEADER
I20260926 07:32:30.089139 23530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } }
I20260926 07:32:30.089645 23520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:30.092197 23531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b3250612e064d93a3a50a95b95ea502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } } }
I20260926 07:32:30.092778 23531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:30.092221 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3250612e064d93a3a50a95b95ea502. Latest consensus state: current_term: 1 leader_uuid: "8b3250612e064d93a3a50a95b95ea502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3250612e064d93a3a50a95b95ea502" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34455 } } }
I20260926 07:32:30.093994 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:30.094892 23536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:30.102385 23536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:30.111258 23536 catalog_manager.cc:1347] Generated new cluster ID: 6393fd4334ef4bfb9beeb86500a8ed76
I20260926 07:32:30.111375 23536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:30.132334 23536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:30.133584 23536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:30.144132 23536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3250612e064d93a3a50a95b95ea502: Generated new TSK 0
I20260926 07:32:30.144939 23536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:30.155397 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23453
W20260926 07:32:30.156565 23515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46167
W20260926 07:32:30.157189 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:30.157353 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23453
2026-09-26T07:32:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:30 dist-test-slave-1dwd krb5kdc[23551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:30 dist-test-slave-1dwd krb5kdc[23551](info): set up 2 sockets
Sep 26 07:32:30 dist-test-slave-1dwd krb5kdc[23551](info): commencing operation
krb5kdc: starting...
W20260926 07:32:31.247743 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:31 dist-test-slave-1dwd krb5kdc[23551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:31.497875 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:31.829470 23564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:31.829774 23564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:31.829942 23564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:31.837414 23564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:31.837532 23564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:31.837630 23564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:31.837746 23564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:31.837822 23564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:31.847748 23564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:31.849128 23564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:31.850556 23564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:31.862403 23569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:31.862468 23572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:31.862452 23570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:31.863471 23564 server_base.cc:1034] running on GCE node
I20260926 07:32:31.864164 23564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:31.865537 23564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:31.866735 23564 hybrid_clock.cc:648] HybridClock initialized: now 1790407951866676 us; error 55 us; skew 500 ppm
I20260926 07:32:31.867074 23564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:31 dist-test-slave-1dwd krb5kdc[23551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:31.870148 23564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:31.871675 23564 webserver.cc:460] Webserver started at http://127.0.0.1:36927/ using document root <none> and password file <none>
I20260926 07:32:31.872354 23564 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:31.872494 23564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:31.872785 23564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:31.875763 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "74779f818c784c739ff9321dddf677e5"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-1dwd"
server_key: "04d5fd1ee9a2f1a78e938ee41477c990"
server_key_iv: "a6d714da9ef4d5fd334acc9931476522"
server_key_version: "encryptionkey@0"
I20260926 07:32:31.876516 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "74779f818c784c739ff9321dddf677e5"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-1dwd"
server_key: "04d5fd1ee9a2f1a78e938ee41477c990"
server_key_iv: "a6d714da9ef4d5fd334acc9931476522"
server_key_version: "encryptionkey@0"
I20260926 07:32:31.881453 23564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:31.884711 23579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:31.885979 23564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:31.886158 23564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "74779f818c784c739ff9321dddf677e5"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-1dwd"
server_key: "04d5fd1ee9a2f1a78e938ee41477c990"
server_key_iv: "a6d714da9ef4d5fd334acc9931476522"
server_key_version: "encryptionkey@0"
I20260926 07:32:31.886356 23564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:31.891884 23564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:31.895593 23564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:31.895876 23564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:31.914108 23564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39873
I20260926 07:32:31.914148 23630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39873 every 8 connection(s)
I20260926 07:32:31.915529 23564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:31.919176 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:31.922176 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23564
I20260926 07:32:31.922375 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:31.922791 2639 external_mini_cluster.cc:1428] Setting key 2effd734c388db8da4b9a4ce3e5de3ba
I20260926 07:32:31.931531 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:31.924554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35175 (local address 127.0.0.1:39873)
0926 07:32:31.925309 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:31.925323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:31.925371 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:32:31.927728 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:31.927760 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:31.928199 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:31.928655 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:31.929628 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:31.929636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:31.929654 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:31.929676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:31.929755 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:31.929779 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:31.930005 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:31.930133 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:32:31.930324 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":312,"threads_started":1}
I20260926 07:32:31.932327 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5: Bootstrap starting.
I20260926 07:32:31.934931 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:31.935572 23631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:31.936525 23631 log.cc:824] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:31.939422 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5: No bootstrap required, opened a new log
I20260926 07:32:31.941444 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:31.938987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57107 (local address 127.0.0.1:39873)
0926 07:32:31.939160 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:31.939165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:31.939539 (+ 374us) server_negotiation.cc:407] Connection header received
0926 07:32:31.939717 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:31.939722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:31.939775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:31.939882 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:31.940760 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:31.940765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:31.940770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:31.940793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:31.940836 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:31.940853 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:31.940979 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:31.941080 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:32:31.941138 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:32:31.946609 23631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260926 07:32:31.946991 23631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:31.947094 23631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74779f818c784c739ff9321dddf677e5, State: Initialized, Role: FOLLOWER
I20260926 07:32:31.947733 23631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260926 07:32:31.947983 23631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:31.948122 23631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:31.948288 23631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:31.950416 23631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260926 07:32:31.950919 23631 leader_election.cc:302] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74779f818c784c739ff9321dddf677e5; no voters:
I20260926 07:32:31.951517 23631 leader_election.cc:288] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:31.951645 23641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:31.953285 23641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [term 1 LEADER]: Becoming Leader. State: Replica: 74779f818c784c739ff9321dddf677e5, State: Running, Role: LEADER
I20260926 07:32:31.953549 23631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:31.953806 23641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } }
I20260926 07:32:31.956346 23643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74779f818c784c739ff9321dddf677e5. Latest consensus state: current_term: 1 leader_uuid: "74779f818c784c739ff9321dddf677e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } } }
I20260926 07:32:31.956749 23643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:31.957290 23642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74779f818c784c739ff9321dddf677e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74779f818c784c739ff9321dddf677e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39873 } } }
I20260926 07:32:31.957620 23642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:31.957790 23649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:31.964550 23649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:31.972576 23649 catalog_manager.cc:1347] Generated new cluster ID: bc397bf3604e48b1b63df02a137142cb
I20260926 07:32:31.972685 23649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:31.991233 23649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:31.991714 23649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:31.999747 23649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74779f818c784c739ff9321dddf677e5: Generated new TSK 0
I20260926 07:32:32.000578 23649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:32.006497 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23564
W20260926 07:32:32.007511 23626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35175
W20260926 07:32:32.008106 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:32.008257 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23564
2026-09-26T07:32:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1850 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:32 dist-test-slave-1dwd krb5kdc[23663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:32 dist-test-slave-1dwd krb5kdc[23663](info): set up 2 sockets
Sep 26 07:32:32 dist-test-slave-1dwd krb5kdc[23663](info): commencing operation
krb5kdc: starting...
W20260926 07:32:33.099543 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:33 dist-test-slave-1dwd krb5kdc[23663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:33.349656 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:33.674474 23676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:33.674801 23676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:33.674961 23676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:33.682175 23676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:33.682317 23676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:33.682406 23676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:33.682497 23676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:33.682574 23676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:33.692353 23676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:33.693606 23676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:33.694885 23676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:33.706245 23684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:33.706245 23682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:33.706480 23681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:33.707195 23676 server_base.cc:1034] running on GCE node
I20260926 07:32:33.707993 23676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:33.709324 23676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:33.710517 23676 hybrid_clock.cc:648] HybridClock initialized: now 1790407953710459 us; error 56 us; skew 500 ppm
I20260926 07:32:33.710850 23676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:33 dist-test-slave-1dwd krb5kdc[23663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:33.717558 23676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:33.718955 23676 webserver.cc:460] Webserver started at http://127.0.0.1:38915/ using document root <none> and password file <none>
I20260926 07:32:33.719719 23676 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:33.719856 23676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:33.720166 23676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:33.723096 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "3c960a12a3ec418eb5c6efb4de75c383"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-1dwd"
server_key: "a1615502c97cff24427dfb4151202e0d"
server_key_iv: "64fd6c4ae7362deee98c16153372d550"
server_key_version: "encryptionkey@0"
I20260926 07:32:33.723918 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "3c960a12a3ec418eb5c6efb4de75c383"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-1dwd"
server_key: "a1615502c97cff24427dfb4151202e0d"
server_key_iv: "64fd6c4ae7362deee98c16153372d550"
server_key_version: "encryptionkey@0"
I20260926 07:32:33.728945 23676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:33.732355 23691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.733702 23676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:33.733891 23676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "3c960a12a3ec418eb5c6efb4de75c383"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-1dwd"
server_key: "a1615502c97cff24427dfb4151202e0d"
server_key_iv: "64fd6c4ae7362deee98c16153372d550"
server_key_version: "encryptionkey@0"
I20260926 07:32:33.734083 23676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:33.771660 23676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:33.775607 23676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:33.775893 23676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:33.793356 23676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44093
I20260926 07:32:33.793388 23742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44093 every 8 connection(s)
I20260926 07:32:33.794684 23676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:33.798285 23743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:33.803572 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23676
I20260926 07:32:33.803738 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:33.804138 2639 external_mini_cluster.cc:1428] Setting key 8b4b7f28e356d50e6857d16b7b0a0427
I20260926 07:32:33.809399 23743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383: Bootstrap starting.
I20260926 07:32:33.812445 23743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:33.813397 23743 log.cc:824] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:33.816231 23743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383: No bootstrap required, opened a new log
Sep 26 07:32:33 dist-test-slave-1dwd krb5kdc[23663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:32:33.824076 23743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260926 07:32:33.824424 23743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:33.824538 23743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c960a12a3ec418eb5c6efb4de75c383, State: Initialized, Role: FOLLOWER
I20260926 07:32:33.825161 23743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260926 07:32:33.825429 23743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:33.825587 23743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:33.825779 23743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:33.826328 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.805833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:59627 (local address 127.0.0.1:44093)
0926 07:32:33.806835 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:33.806853 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:33.806913 (+ 60us) server_negotiation.cc:407] Connection header received
0926 07:32:33.808795 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:33.808820 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:33.809195 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:33.809573 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:33.811009 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.812334 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.813590 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.814259 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.814463 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.818874 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:33.818881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:33.818894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:33.818936 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:33.821998 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.822786 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.822791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.822800 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.822965 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.823503 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.823506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.823508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.823972 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:33.824290 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:33.824770 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 07:32:33.825105 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":450,"threads_started":1}
I20260926 07:32:33.828349 23743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260926 07:32:33.828859 23743 leader_election.cc:302] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c960a12a3ec418eb5c6efb4de75c383; no voters:
I20260926 07:32:33.829484 23743 leader_election.cc:288] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:33.829612 23748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:33.830451 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:33.831341 23743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:33.836966 23748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [term 1 LEADER]: Becoming Leader. State: Replica: 3c960a12a3ec418eb5c6efb4de75c383, State: Running, Role: LEADER
I20260926 07:32:33.837778 23748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } }
I20260926 07:32:33.848248 23749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c960a12a3ec418eb5c6efb4de75c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } } }
I20260926 07:32:33.848702 23749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:33.849279 23750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c960a12a3ec418eb5c6efb4de75c383. Latest consensus state: current_term: 1 leader_uuid: "3c960a12a3ec418eb5c6efb4de75c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c960a12a3ec418eb5c6efb4de75c383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44093 } } }
I20260926 07:32:33.849610 23750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:33.853855 23765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:33.857681 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.838072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:36307 (local address 127.0.0.1:44093)
0926 07:32:33.838302 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:33.838310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:33.838334 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:32:33.838400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:33.838407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:33.838501 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:33.838625 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:33.841031 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.842584 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.845096 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.846086 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.846238 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.849835 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:33.849844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:33.849853 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:33.849915 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:33.854401 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.855321 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.855328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.855332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.855468 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.855984 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.855988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.855992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.856424 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:33.856706 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:33.857096 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 07:32:33.857264 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:32:33.859773 23765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:33.868896 23765 catalog_manager.cc:1347] Generated new cluster ID: 23b19dc823fd4549975618d2144b6d9c
I20260926 07:32:33.868997 23765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:33.896266 23765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:33.898301 23765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:33.916183 23765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c960a12a3ec418eb5c6efb4de75c383: Generated new TSK 0
I20260926 07:32:33.917245 23765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:33.924594 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23676
I20260926 07:32:33.925959 23738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:34.017434 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23676
W20260926 07:32:34.034361 23661 connection.cc:535] client connection to 127.0.0.1:44093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44093): Connection reset by peer (error 104)
2026-09-26T07:32:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:34 dist-test-slave-1dwd krb5kdc[23776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:34 dist-test-slave-1dwd krb5kdc[23776](info): set up 2 sockets
Sep 26 07:32:34 dist-test-slave-1dwd krb5kdc[23776](info): commencing operation
krb5kdc: starting...
W20260926 07:32:35.139968 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:35 dist-test-slave-1dwd krb5kdc[23776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:35.406265 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:35.737514 23789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:35.737792 23789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:35.737891 23789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:35.745570 23789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:35.745675 23789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:35.745729 23789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:35.745790 23789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:35.745833 23789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:35.755446 23789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:35.756875 23789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:35.758338 23789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:35.769951 23794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:35.769914 23795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:35.770005 23797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:35.770785 23789 server_base.cc:1034] running on GCE node
I20260926 07:32:35.771610 23789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:35.772956 23789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:35.774175 23789 hybrid_clock.cc:648] HybridClock initialized: now 1790407955774131 us; error 39 us; skew 500 ppm
I20260926 07:32:35.774518 23789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:35 dist-test-slave-1dwd krb5kdc[23776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:35.777587 23789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:35.779063 23789 webserver.cc:460] Webserver started at http://127.0.0.1:33945/ using document root <none> and password file <none>
I20260926 07:32:35.779803 23789 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:35.779950 23789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:35.780242 23789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:35.783315 23789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "15b6cb59d1a3489aaa0816248aa0b0a2"
format_stamp: "Formatted at 2026-09-26 07:32:35 on dist-test-slave-1dwd"
server_key: "64bf869bfda8ca6d0af31f9fa19460ff"
server_key_iv: "01d6820a4eae876be66423fff8e50137"
server_key_version: "encryptionkey@0"
I20260926 07:32:35.784366 23789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "15b6cb59d1a3489aaa0816248aa0b0a2"
format_stamp: "Formatted at 2026-09-26 07:32:35 on dist-test-slave-1dwd"
server_key: "64bf869bfda8ca6d0af31f9fa19460ff"
server_key_iv: "01d6820a4eae876be66423fff8e50137"
server_key_version: "encryptionkey@0"
I20260926 07:32:35.789438 23789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 07:32:35.792680 23804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:35.793957 23789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 07:32:35.794150 23789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "15b6cb59d1a3489aaa0816248aa0b0a2"
format_stamp: "Formatted at 2026-09-26 07:32:35 on dist-test-slave-1dwd"
server_key: "64bf869bfda8ca6d0af31f9fa19460ff"
server_key_iv: "01d6820a4eae876be66423fff8e50137"
server_key_version: "encryptionkey@0"
I20260926 07:32:35.794330 23789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:35.808583 23789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:35.812570 23789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:35.812860 23789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:35.830938 23789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37083
I20260926 07:32:35.830979 23855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37083 every 8 connection(s)
I20260926 07:32:35.832327 23789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:35.835983 23856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:35.840392 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23789
I20260926 07:32:35.840565 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:35.840974 2639 external_mini_cluster.cc:1428] Setting key 4e95acb1d782e04720d935b58bbe4ad5
I20260926 07:32:35.848292 23856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2: Bootstrap starting.
I20260926 07:32:35.851878 23856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:35.853194 23856 log.cc:824] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:35.856204 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.842725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:43845 (local address 127.0.0.1:37083)
0926 07:32:35.843539 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.843555 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.843609 (+ 54us) server_negotiation.cc:407] Connection header received
0926 07:32:35.846469 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.846502 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.846986 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.847450 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:35.847947 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.849912 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.851197 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.852311 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.852520 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.853715 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:35.853729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:35.853752 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:35.854342 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:35.854449 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:35.854479 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:35.854631 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.854851 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.855003 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":342,"threads_started":1}
I20260926 07:32:35.856396 23856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2: No bootstrap required, opened a new log
I20260926 07:32:35.859913 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:35.864606 23856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260926 07:32:35.865001 23856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:35.865130 23856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b6cb59d1a3489aaa0816248aa0b0a2, State: Initialized, Role: FOLLOWER
I20260926 07:32:35.865756 23856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260926 07:32:35.866032 23856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:35.866190 23856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:35.866379 23856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:35.868749 23856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260926 07:32:35.869308 23856 leader_election.cc:302] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b6cb59d1a3489aaa0816248aa0b0a2; no voters:
I20260926 07:32:35.870035 23856 leader_election.cc:288] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:35.870154 23866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:35.871150 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.864352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:36983 (local address 127.0.0.1:37083)
0926 07:32:35.864528 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.864534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.864928 (+ 394us) server_negotiation.cc:407] Connection header received
0926 07:32:35.865131 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.865137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.865217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.865333 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:35.865813 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.866781 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.868172 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.868641 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.868771 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.869973 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:35.869978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:35.869981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:35.869998 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:35.870060 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:35.870076 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:35.870201 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.870791 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.870843 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:32:35.871778 23866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 15b6cb59d1a3489aaa0816248aa0b0a2, State: Running, Role: LEADER
I20260926 07:32:35.872632 23856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:35.872445 23866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260926 07:32:35.875687 23868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b6cb59d1a3489aaa0816248aa0b0a2. Latest consensus state: current_term: 1 leader_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260926 07:32:35.876194 23868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:35.876768 23867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b6cb59d1a3489aaa0816248aa0b0a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260926 07:32:35.877187 23874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:35.877707 23867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:35.883293 23874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:35.891861 23874 catalog_manager.cc:1347] Generated new cluster ID: e800991074ec46bb8dc601053373361b
I20260926 07:32:35.891981 23874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:35.904748 23874 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:35.906581 23874 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:35.918886 23874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b6cb59d1a3489aaa0816248aa0b0a2: Generated new TSK 0
I20260926 07:32:35.919956 23874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:35.926756 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23789
W20260926 07:32:35.927978 23851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:43845
W20260926 07:32:35.928746 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23789: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:35.928905 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23789
W20260926 07:32:35.945237 23774 connection.cc:535] client connection to 127.0.0.1:37083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37083): Connection reset by peer (error 104)
2026-09-26T07:32:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:36 dist-test-slave-1dwd krb5kdc[23887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:36 dist-test-slave-1dwd krb5kdc[23887](info): set up 2 sockets
Sep 26 07:32:36 dist-test-slave-1dwd krb5kdc[23887](info): commencing operation
krb5kdc: starting...
W20260926 07:32:37.027747 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:37 dist-test-slave-1dwd krb5kdc[23887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:37.277614 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:37.606616 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:37.606933 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:37.607093 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:37.614187 23900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:37.614327 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:37.614428 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:37.614529 23900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:37.614609 23900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:37.624289 23900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:37.625540 23900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:37.626823 23900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:37.638619 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:37.638671 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:37.638671 23908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:37.639524 23900 server_base.cc:1034] running on GCE node
I20260926 07:32:37.640304 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:37.641573 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:37.642861 23900 hybrid_clock.cc:648] HybridClock initialized: now 1790407957642811 us; error 36 us; skew 500 ppm
I20260926 07:32:37.643338 23900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:37 dist-test-slave-1dwd krb5kdc[23887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:37.647763 23900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:37.651106 23900 webserver.cc:460] Webserver started at http://127.0.0.1:39497/ using document root <none> and password file <none>
I20260926 07:32:37.652015 23900 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:37.652119 23900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:37.652364 23900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:37.655186 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "23118ba1e95f457c8b4fe7b0b368e54e"
format_stamp: "Formatted at 2026-09-26 07:32:37 on dist-test-slave-1dwd"
server_key: "71359d5530ed6e1bead4123d25380662"
server_key_iv: "91dcfce6210c112a02410ed9454da735"
server_key_version: "encryptionkey@0"
I20260926 07:32:37.655915 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "23118ba1e95f457c8b4fe7b0b368e54e"
format_stamp: "Formatted at 2026-09-26 07:32:37 on dist-test-slave-1dwd"
server_key: "71359d5530ed6e1bead4123d25380662"
server_key_iv: "91dcfce6210c112a02410ed9454da735"
server_key_version: "encryptionkey@0"
I20260926 07:32:37.660715 23900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:37.664113 23915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:37.665361 23900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:37.665531 23900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "23118ba1e95f457c8b4fe7b0b368e54e"
format_stamp: "Formatted at 2026-09-26 07:32:37 on dist-test-slave-1dwd"
server_key: "71359d5530ed6e1bead4123d25380662"
server_key_iv: "91dcfce6210c112a02410ed9454da735"
server_key_version: "encryptionkey@0"
I20260926 07:32:37.665719 23900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:37.671275 23900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:37.674930 23900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:37.675252 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:37.692332 23900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36287
I20260926 07:32:37.692368 23966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36287 every 8 connection(s)
I20260926 07:32:37.693699 23900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:37.697242 23967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:37.701025 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 23900
I20260926 07:32:37.701237 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:37.701757 2639 external_mini_cluster.cc:1428] Setting key 5b1fb77f1ac74431c0fe38170f122c48
I20260926 07:32:37.709141 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:37.703570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:54913 (local address 127.0.0.1:36287)
0926 07:32:37.704328 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:37.704342 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:37.704387 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:32:37.705943 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:37.705971 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:37.706307 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:37.706628 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:37.707687 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:37.707693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:37.707705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:37.707719 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:37.707778 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:37.707796 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:37.707962 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:37.708079 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:32:37.708279 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":357,"threads_started":1}
I20260926 07:32:37.709812 23967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e: Bootstrap starting.
I20260926 07:32:37.713009 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:37.713513 23967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:37.714718 23967 log.cc:824] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:37.718046 23967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e: No bootstrap required, opened a new log
I20260926 07:32:37.720639 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:37.718082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:45633 (local address 127.0.0.1:36287)
0926 07:32:37.718226 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:37.718231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:37.718762 (+ 531us) server_negotiation.cc:407] Connection header received
0926 07:32:37.718936 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:37.718940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:37.718994 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:37.719072 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:37.719990 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:37.719998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:37.720001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:37.720015 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:37.720050 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:37.720067 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:37.720159 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:37.720290 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:32:37.720335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:32:37.726511 23967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260926 07:32:37.726873 23967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:37.726974 23967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23118ba1e95f457c8b4fe7b0b368e54e, State: Initialized, Role: FOLLOWER
I20260926 07:32:37.727608 23967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260926 07:32:37.727849 23967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:37.727994 23967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:37.728160 23967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:37.730274 23967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260926 07:32:37.730747 23967 leader_election.cc:302] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23118ba1e95f457c8b4fe7b0b368e54e; no voters:
I20260926 07:32:37.731380 23967 leader_election.cc:288] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:37.731513 23977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:37.732178 23977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [term 1 LEADER]: Becoming Leader. State: Replica: 23118ba1e95f457c8b4fe7b0b368e54e, State: Running, Role: LEADER
I20260926 07:32:37.732772 23977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } }
I20260926 07:32:37.733541 23967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:37.735519 23979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23118ba1e95f457c8b4fe7b0b368e54e. Latest consensus state: current_term: 1 leader_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } } }
I20260926 07:32:37.735447 23978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23118ba1e95f457c8b4fe7b0b368e54e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36287 } } }
I20260926 07:32:37.736001 23979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:37.736058 23978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:37.737578 23984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:37.743271 23984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:37.751973 23984 catalog_manager.cc:1347] Generated new cluster ID: 66dd81f6b81d4895b1f70a7d2e3cc1f7
I20260926 07:32:37.752084 23984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:37.797806 23984 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:37.798264 23984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:37.806308 23984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23118ba1e95f457c8b4fe7b0b368e54e: Generated new TSK 0
I20260926 07:32:37.807106 23984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:37.819707 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23900
W20260926 07:32:37.820791 23962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:54913
W20260926 07:32:37.821427 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 23900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:37.821578 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 23900
2026-09-26T07:32:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:37 dist-test-slave-1dwd krb5kdc[23998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:37 dist-test-slave-1dwd krb5kdc[23998](info): set up 2 sockets
Sep 26 07:32:37 dist-test-slave-1dwd krb5kdc[23998](info): commencing operation
krb5kdc: starting...
W20260926 07:32:38.955528 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[23998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:39.194217 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:39.520213 24011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:39.520496 24011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:39.520653 24011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:39.527854 24011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:39.527962 24011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:39.528091 24011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:39.528194 24011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:39.528278 24011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:39.538069 24011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:39.539386 24011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:39.540715 24011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:39.552651 24019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:39.552699 24016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:39.552728 24017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:39.553373 24011 server_base.cc:1034] running on GCE node
I20260926 07:32:39.554169 24011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:39.555567 24011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:39.556764 24011 hybrid_clock.cc:648] HybridClock initialized: now 1790407959556715 us; error 39 us; skew 500 ppm
I20260926 07:32:39.557120 24011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[23998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:39.563931 24011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:39.565411 24011 webserver.cc:460] Webserver started at http://127.0.0.1:46365/ using document root <none> and password file <none>
I20260926 07:32:39.566146 24011 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:39.566273 24011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:39.566593 24011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:39.569795 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "f2d938de777242d19c7aeb0e75157012"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-1dwd"
server_key: "e629ed7cf05bfb4e8ee213592ce7afba"
server_key_iv: "bdbdd0819eff1fa90365d6e825b8c933"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.570545 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "f2d938de777242d19c7aeb0e75157012"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-1dwd"
server_key: "e629ed7cf05bfb4e8ee213592ce7afba"
server_key_iv: "bdbdd0819eff1fa90365d6e825b8c933"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.575625 24011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 07:32:39.578902 24026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:39.580211 24011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:39.580381 24011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f2d938de777242d19c7aeb0e75157012"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-1dwd"
server_key: "e629ed7cf05bfb4e8ee213592ce7afba"
server_key_iv: "bdbdd0819eff1fa90365d6e825b8c933"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.580588 24011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:39.610756 24011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:39.615363 24011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:39.615684 24011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:39.634116 24011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38587
I20260926 07:32:39.634136 24077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38587 every 8 connection(s)
I20260926 07:32:39.635504 24011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:39.637275 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24011
I20260926 07:32:39.637460 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:39.637900 2639 external_mini_cluster.cc:1428] Setting key cc03c756da71d164a4c8397306cd8590
I20260926 07:32:39.639647 24078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[23998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:32:39.652976 24078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012: Bootstrap starting.
I20260926 07:32:39.657048 24078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:39.658298 24078 log.cc:824] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:39.661295 24078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012: No bootstrap required, opened a new log
I20260926 07:32:39.661741 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.639840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:42203 (local address 127.0.0.1:38587)
0926 07:32:39.640609 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.640623 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.640670 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:32:39.642916 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.642943 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.643506 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.643952 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.645262 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.647327 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.648503 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.649501 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.649683 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.653674 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.653684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.653701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.653744 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.657719 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.658420 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.658428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.658435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.658624 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.659066 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.659073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.659081 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.659633 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.659952 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.660437 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.660738 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":293,"threads_started":1}
I20260926 07:32:39.665427 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:39.669361 24078 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 07:32:39.669792 24078 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:39.669906 24078 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d938de777242d19c7aeb0e75157012, State: Initialized, Role: FOLLOWER
I20260926 07:32:39.670655 24078 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 07:32:39.670970 24078 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:39.671123 24078 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:39.671370 24078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:39.674217 24078 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 07:32:39.674798 24078 leader_election.cc:302] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2d938de777242d19c7aeb0e75157012; no voters:
I20260926 07:32:39.675489 24078 leader_election.cc:288] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:39.675611 24088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:39.676908 24088 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [term 1 LEADER]: Becoming Leader. State: Replica: f2d938de777242d19c7aeb0e75157012, State: Running, Role: LEADER
I20260926 07:32:39.677587 24088 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } }
I20260926 07:32:39.677769 24078 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:39.680646 24090 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2d938de777242d19c7aeb0e75157012. Latest consensus state: current_term: 1 leader_uuid: "f2d938de777242d19c7aeb0e75157012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } } }
I20260926 07:32:39.681074 24090 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:39.681597 24089 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2d938de777242d19c7aeb0e75157012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d938de777242d19c7aeb0e75157012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38587 } } }
I20260926 07:32:39.681875 24089 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:39.689258 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.669876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:45475 (local address 127.0.0.1:38587)
0926 07:32:39.670114 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.670122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.670484 (+ 362us) server_negotiation.cc:407] Connection header received
0926 07:32:39.670653 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.670656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.670713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.670788 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.672744 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.673508 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.675360 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.676082 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.677271 (+ 1189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.679689 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.679696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.679702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.679744 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.683936 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.685167 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.685176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.685181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.685497 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.686144 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.686152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.686158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.686644 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.686994 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.688521 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.688754 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:39.690955 24095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:39.696179 24095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:39.704759 24095 catalog_manager.cc:1347] Generated new cluster ID: c09bae74fef94eb2944e5ef925f7f58f
I20260926 07:32:39.704869 24095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:39.736692 24095 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:39.738020 24095 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:39.746546 24095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2d938de777242d19c7aeb0e75157012: Generated new TSK 0
I20260926 07:32:39.747475 24095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:39.759984 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24011
I20260926 07:32:39.761461 24073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:39.853880 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24011
W20260926 07:32:39.871160 23996 connection.cc:535] client connection to 127.0.0.1:38587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38587): Connection reset by peer (error 104)
2026-09-26T07:32:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2032 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[24111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[24111](info): set up 2 sockets
Sep 26 07:32:39 dist-test-slave-1dwd krb5kdc[24111](info): commencing operation
krb5kdc: starting...
W20260926 07:32:40.943688 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:41 dist-test-slave-1dwd krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:41.191229 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:41.511864 24124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:41.512157 24124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:41.512327 24124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:41.519440 24124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:41.519541 24124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:41.519650 24124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:41.519753 24124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:41.519846 24124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:41.529392 24124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:41.530671 24124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:41.532048 24124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:41.543772 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:41.543767 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:41.543845 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:41.544890 24124 server_base.cc:1034] running on GCE node
I20260926 07:32:41.545626 24124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:41.546977 24124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:41.548200 24124 hybrid_clock.cc:648] HybridClock initialized: now 1790407961548136 us; error 59 us; skew 500 ppm
I20260926 07:32:41.548547 24124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:41 dist-test-slave-1dwd krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:41.551560 24124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:41.553181 24124 webserver.cc:460] Webserver started at http://127.0.0.1:39899/ using document root <none> and password file <none>
I20260926 07:32:41.553836 24124 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:41.553987 24124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:41.554291 24124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:41.557291 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "87f2c80e916d4235b7acf6bea039af89"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-1dwd"
server_key: "827ed60ee1d4af10939f82966c2b1e46"
server_key_iv: "3a766a9afe933e881c6c62fbe1032dfe"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.558050 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "87f2c80e916d4235b7acf6bea039af89"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-1dwd"
server_key: "827ed60ee1d4af10939f82966c2b1e46"
server_key_iv: "3a766a9afe933e881c6c62fbe1032dfe"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.563072 24124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 07:32:41.566464 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:41.567775 24124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:41.567970 24124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "87f2c80e916d4235b7acf6bea039af89"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-1dwd"
server_key: "827ed60ee1d4af10939f82966c2b1e46"
server_key_iv: "3a766a9afe933e881c6c62fbe1032dfe"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.568167 24124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:41.594620 24124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:41.598791 24124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:41.599097 24124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:41.617328 24124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41711
I20260926 07:32:41.617369 24190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41711 every 8 connection(s)
I20260926 07:32:41.618708 24124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:41.622294 24191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:41.626565 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24124
I20260926 07:32:41.626752 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:41.627322 2639 external_mini_cluster.cc:1428] Setting key a854fc24cbfe853ab9b5a8bc4601346c
I20260926 07:32:41.634711 24191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89: Bootstrap starting.
I20260926 07:32:41.638360 24191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:41.639719 24191 log.cc:824] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:41.642943 24191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89: No bootstrap required, opened a new log
I20260926 07:32:41.643143 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:41.629441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:48395 (local address 127.0.0.1:41711)
0926 07:32:41.630370 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:41.630386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:41.630755 (+ 369us) server_negotiation.cc:407] Connection header received
0926 07:32:41.633298 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:41.633328 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:41.633846 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:41.634288 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:41.634798 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.636791 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.638061 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.639140 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.639373 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:41.640576 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:41.640586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:41.640599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:41.641229 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:41.641316 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:41.641346 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:41.641488 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:41.641738 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:32:41.641907 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":396,"threads_started":1}
I20260926 07:32:41.646770 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:41.652688 24191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260926 07:32:41.653219 24191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:41.653319 24191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f2c80e916d4235b7acf6bea039af89, State: Initialized, Role: FOLLOWER
I20260926 07:32:41.653973 24191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260926 07:32:41.654227 24191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:41.654378 24191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:41.654554 24191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:41.656863 24191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260926 07:32:41.657423 24191 leader_election.cc:302] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f2c80e916d4235b7acf6bea039af89; no voters:
I20260926 07:32:41.658185 24191 leader_election.cc:288] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:41.658331 24201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:41.658470 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:41.651996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:49819 (local address 127.0.0.1:41711)
0926 07:32:41.652168 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:41.652174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:41.652615 (+ 441us) server_negotiation.cc:407] Connection header received
0926 07:32:41.652759 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:41.652764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:41.652837 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:41.652968 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:41.653523 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.654342 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.655733 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.656195 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.656341 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:41.657445 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:41.657450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:41.657453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:41.657470 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:41.657521 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:41.657535 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:41.657658 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:41.657847 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:32:41.657909 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:32:41.659122 24201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [term 1 LEADER]: Becoming Leader. State: Replica: 87f2c80e916d4235b7acf6bea039af89, State: Running, Role: LEADER
I20260926 07:32:41.659770 24201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260926 07:32:41.661087 24191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:41.662575 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f2c80e916d4235b7acf6bea039af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } } }
I20260926 07:32:41.662564 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f2c80e916d4235b7acf6bea039af89. Latest consensus state: current_term: 1 leader_uuid: "87f2c80e916d4235b7acf6bea039af89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f2c80e916d4235b7acf6bea039af89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } } }
I20260926 07:32:41.663007 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:41.663007 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:41.663868 24206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:41.668928 24206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:41.679538 24206 catalog_manager.cc:1347] Generated new cluster ID: c1e67be2b8744b65b68b792c4c79acfb
I20260926 07:32:41.679679 24206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:41.694005 24206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:41.695382 24206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:41.705665 24206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f2c80e916d4235b7acf6bea039af89: Generated new TSK 0
I20260926 07:32:41.706564 24206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:41.722079 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24124
W20260926 07:32:41.724119 24186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:48395
W20260926 07:32:41.724998 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24124: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:41.725226 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24124
W20260926 07:32:41.742224 24109 connection.cc:535] client connection to 127.0.0.1:41711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41711): Connection reset by peer (error 104)
2026-09-26T07:32:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:41 dist-test-slave-1dwd krb5kdc[24222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:41 dist-test-slave-1dwd krb5kdc[24222](info): set up 2 sockets
Sep 26 07:32:41 dist-test-slave-1dwd krb5kdc[24222](info): commencing operation
krb5kdc: starting...
W20260926 07:32:42.811748 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:42 dist-test-slave-1dwd krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:32:43.050320 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:43.369480 24235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:43.369731 24235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:43.369859 24235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:43.376978 24235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:43.377074 24235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:43.377125 24235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:43.377213 24235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:43.377277 24235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:43.386730 24235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:43.388044 24235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:43.389346 24235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:43.400768 24240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.400868 24241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.401221 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.401682 24235 server_base.cc:1034] running on GCE node
I20260926 07:32:43.402513 24235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:43.404011 24235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:43.405239 24235 hybrid_clock.cc:648] HybridClock initialized: now 1790407963405200 us; error 67 us; skew 500 ppm
I20260926 07:32:43.405586 24235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:43 dist-test-slave-1dwd krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:43.408670 24235 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:32:43.410027 24235 webserver.cc:460] Webserver started at http://127.0.0.1:38991/ using document root <none> and password file <none>
I20260926 07:32:43.410710 24235 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:43.410856 24235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:43.411154 24235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:43.414121 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "ea0fcd815b6b4eba997a935bc8e30506"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-1dwd"
server_key: "e17591446dbd68422668a6def47b6c3b"
server_key_iv: "5cc08d02673299e983e4faf55a78dedf"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.414875 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "ea0fcd815b6b4eba997a935bc8e30506"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-1dwd"
server_key: "e17591446dbd68422668a6def47b6c3b"
server_key_iv: "5cc08d02673299e983e4faf55a78dedf"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.419775 24235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:43.423039 24250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.424322 24235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:43.424525 24235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "ea0fcd815b6b4eba997a935bc8e30506"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-1dwd"
server_key: "e17591446dbd68422668a6def47b6c3b"
server_key_iv: "5cc08d02673299e983e4faf55a78dedf"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.424718 24235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:43.430197 24235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:43.434074 24235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:43.434370 24235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:43.451408 24235 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37749
I20260926 07:32:43.451457 24301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37749 every 8 connection(s)
I20260926 07:32:43.452847 24235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:43.453233 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24235
I20260926 07:32:43.453420 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:43.454044 2639 external_mini_cluster.cc:1428] Setting key cb5fbb6e479742680c428cf4de514611
I20260926 07:32:43.457471 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:43.462814 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.455946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:54761 (local address 127.0.0.1:37749)
0926 07:32:43.456711 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:43.456725 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:43.456771 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:32:43.459070 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:43.459096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:43.459556 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:43.459975 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:43.460941 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:43.460950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:43.460965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:43.460984 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:43.461077 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:43.461101 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:43.461314 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:43.461449 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:32:43.461653 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
I20260926 07:32:43.466508 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:43.470055 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506: Bootstrap starting.
I20260926 07:32:43.473150 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.470821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:38477 (local address 127.0.0.1:37749)
0926 07:32:43.470979 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:43.470985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:43.471368 (+ 383us) server_negotiation.cc:407] Connection header received
0926 07:32:43.471503 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:43.471513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:43.471583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:43.471721 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:43.472556 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:43.472563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:43.472567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:43.472585 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:43.472629 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:43.472650 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:43.472765 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:43.472838 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:32:43.472893 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:32:43.473644 24302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:43.474888 24302 log.cc:824] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:43.478348 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506: No bootstrap required, opened a new log
I20260926 07:32:43.485553 24302 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260926 07:32:43.485926 24302 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:43.486040 24302 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0fcd815b6b4eba997a935bc8e30506, State: Initialized, Role: FOLLOWER
I20260926 07:32:43.486586 24302 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260926 07:32:43.486804 24302 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:43.486969 24302 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:43.487131 24302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:43.489250 24302 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260926 07:32:43.489730 24302 leader_election.cc:302] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea0fcd815b6b4eba997a935bc8e30506; no voters:
I20260926 07:32:43.490437 24302 leader_election.cc:288] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:43.490608 24312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:43.492265 24312 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [term 1 LEADER]: Becoming Leader. State: Replica: ea0fcd815b6b4eba997a935bc8e30506, State: Running, Role: LEADER
I20260926 07:32:43.492815 24302 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:43.492774 24312 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } }
I20260926 07:32:43.496671 24314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea0fcd815b6b4eba997a935bc8e30506. Latest consensus state: current_term: 1 leader_uuid: "ea0fcd815b6b4eba997a935bc8e30506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } } }
I20260926 07:32:43.497095 24314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:43.497629 24313 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea0fcd815b6b4eba997a935bc8e30506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea0fcd815b6b4eba997a935bc8e30506" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37749 } } }
I20260926 07:32:43.497910 24313 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:43.498106 24322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:43.504714 24322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:43.514683 24322 catalog_manager.cc:1347] Generated new cluster ID: 19f3f4dd088d46d68ec85deb9a46996b
I20260926 07:32:43.514808 24322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:43.533823 24322 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:43.534252 24322 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:43.540496 24322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea0fcd815b6b4eba997a935bc8e30506: Generated new TSK 0
I20260926 07:32:43.541211 24322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:43.551971 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24235
W20260926 07:32:43.552977 24297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:54761
W20260926 07:32:43.553644 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:43.553800 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24235
2026-09-26T07:32:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:43 dist-test-slave-1dwd krb5kdc[24333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:43 dist-test-slave-1dwd krb5kdc[24333](info): set up 2 sockets
Sep 26 07:32:43 dist-test-slave-1dwd krb5kdc[24333](info): commencing operation
krb5kdc: starting...
W20260926 07:32:44.643694 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:44 dist-test-slave-1dwd krb5kdc[24333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:44.882417 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38023
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:45.208406 24346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:45.208673 24346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:45.208822 24346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:45.215794 24346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:45.215911 24346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:45.216029 24346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:45.216135 24346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:45.216183 24346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:45.225373 24346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:45.226637 24346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:45.227977 24346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:45.242482 24351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.242522 24352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.243350 24354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:45.244762 24346 server_base.cc:1034] running on GCE node
I20260926 07:32:45.245853 24346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:45.247716 24346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:45.248924 24346 hybrid_clock.cc:648] HybridClock initialized: now 1790407965248866 us; error 49 us; skew 500 ppm
I20260926 07:32:45.249336 24346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:45 dist-test-slave-1dwd krb5kdc[24333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:45.252977 24346 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:45.254675 24346 webserver.cc:460] Webserver started at http://127.2.147.254:40617/ using document root <none> and password file <none>
I20260926 07:32:45.255672 24346 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:45.255851 24346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:45.256223 24346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:45.260723 24346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "f0a8d05ca8074482bd3779e4273754f1"
format_stamp: "Formatted at 2026-09-26 07:32:45 on dist-test-slave-1dwd"
server_key: "e534a175d4c56d90d440366231480133"
server_key_iv: "b4deeb840111849064953fd176b08ded"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.261664 24346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "f0a8d05ca8074482bd3779e4273754f1"
format_stamp: "Formatted at 2026-09-26 07:32:45 on dist-test-slave-1dwd"
server_key: "e534a175d4c56d90d440366231480133"
server_key_iv: "b4deeb840111849064953fd176b08ded"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.268648 24346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 07:32:45.273720 24361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.275151 24346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 07:32:45.275377 24346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "f0a8d05ca8074482bd3779e4273754f1"
format_stamp: "Formatted at 2026-09-26 07:32:45 on dist-test-slave-1dwd"
server_key: "e534a175d4c56d90d440366231480133"
server_key_iv: "b4deeb840111849064953fd176b08ded"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.275687 24346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:45.299412 24346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:45.303171 24346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:45.303514 24346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:45.321194 24346 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38023
I20260926 07:32:45.321214 24412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38023 every 8 connection(s)
I20260926 07:32:45.322500 24346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:45.326066 24413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:45.326398 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24346
I20260926 07:32:45.326571 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:45.327128 2639 external_mini_cluster.cc:1428] Setting key cf1e8b5ffeef47bafe6a1c481b622b19
I20260926 07:32:45.340065 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1: Bootstrap starting.
Sep 26 07:32:45 dist-test-slave-1dwd krb5kdc[24333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:45.343755 24413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:45.344872 24413 log.cc:824] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:45.347858 24413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1: No bootstrap required, opened a new log
I20260926 07:32:45.350685 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.328847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45193 (local address 127.2.147.254:38023)
0926 07:32:45.329676 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.329687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.329728 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:45.332151 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.332174 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.332585 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.332914 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:45.334648 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.335958 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.337856 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.338509 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.338643 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.343491 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:45.343498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:45.343512 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:45.343551 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:45.346669 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.347458 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.347464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.347469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.347619 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.348097 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.348107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.348109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.348602 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:45.348938 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.349445 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.349723 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":397,"threads_started":1}
I20260926 07:32:45.354606 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:45.356421 24413 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } }
I20260926 07:32:45.356779 24413 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:45.356889 24413 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a8d05ca8074482bd3779e4273754f1, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.357556 24413 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } }
I20260926 07:32:45.357832 24413 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:45.357985 24413 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:45.358156 24413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:45.361400 24413 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } }
I20260926 07:32:45.362011 24413 leader_election.cc:302] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a8d05ca8074482bd3779e4273754f1; no voters:
I20260926 07:32:45.362679 24413 leader_election.cc:288] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:45.362807 24423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:45.363914 24423 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [term 1 LEADER]: Becoming Leader. State: Replica: f0a8d05ca8074482bd3779e4273754f1, State: Running, Role: LEADER
I20260926 07:32:45.364485 24423 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } }
I20260926 07:32:45.365155 24413 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:45.367388 24425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a8d05ca8074482bd3779e4273754f1. Latest consensus state: current_term: 1 leader_uuid: "f0a8d05ca8074482bd3779e4273754f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } } }
I20260926 07:32:45.367776 24425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:45.368525 24424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a8d05ca8074482bd3779e4273754f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a8d05ca8074482bd3779e4273754f1" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38023 } } }
I20260926 07:32:45.368840 24424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:45.379220 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.359144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46859 (local address 127.2.147.254:38023)
0926 07:32:45.359393 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.359400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.359664 (+ 264us) server_negotiation.cc:407] Connection header received
0926 07:32:45.359821 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.359825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.359889 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.359974 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:45.361836 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.362633 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.364426 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.365018 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.365161 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.369210 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:45.369219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:45.369227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:45.369278 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:45.373970 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.375241 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.375250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.375256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.375439 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.376114 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.376122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.376127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.376655 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:45.378259 (+ 1604us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.378501 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.378714 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 07:32:45.381193 24429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:45.386099 24429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:45.394407 24429 catalog_manager.cc:1347] Generated new cluster ID: e83d356add974630b858d972005f1bb0
I20260926 07:32:45.394522 24429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:45.409617 24429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:45.411484 24429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:45.423796 24429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a8d05ca8074482bd3779e4273754f1: Generated new TSK 0
I20260926 07:32:45.424714 24429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:45.440546 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24346
I20260926 07:32:45.441715 24408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:45.533850 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24346
W20260926 07:32:45.550974 24331 connection.cc:535] client connection to 127.2.147.254:38023 recv error: Network error: failed to read from TLS socket (remote: 127.2.147.254:38023): Connection reset by peer (error 104)
2026-09-26T07:32:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:45 dist-test-slave-1dwd krb5kdc[24445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:45 dist-test-slave-1dwd krb5kdc[24445](info): set up 2 sockets
Sep 26 07:32:45 dist-test-slave-1dwd krb5kdc[24445](info): commencing operation
krb5kdc: starting...
W20260926 07:32:46.627698 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:46 dist-test-slave-1dwd krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:46.867928 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41187
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:47.193874 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:47.194152 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:47.194244 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:47.201093 24458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:47.201190 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:47.201241 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:47.201298 24458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:47.201341 24458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:47.210652 24458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:47.211912 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:47.213140 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:47.224622 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:47.224572 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:47.224658 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:47.225736 24458 server_base.cc:1034] running on GCE node
I20260926 07:32:47.226444 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:47.227782 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:47.228973 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790407967228919 us; error 51 us; skew 500 ppm
I20260926 07:32:47.229319 24458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:47 dist-test-slave-1dwd krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407967, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:47.232415 24458 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:47.233973 24458 webserver.cc:460] Webserver started at http://127.2.147.254:43691/ using document root <none> and password file <none>
I20260926 07:32:47.234865 24458 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:47.235010 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:47.235378 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:47.238303 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "6bab28c52ad943f49495f65a9bf6b449"
format_stamp: "Formatted at 2026-09-26 07:32:47 on dist-test-slave-1dwd"
server_key: "58f5551bb66ee32cb77dd713f60b5c45"
server_key_iv: "b8a0f04f644d33ed32997feddcf46064"
server_key_version: "encryptionkey@0"
I20260926 07:32:47.239053 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "6bab28c52ad943f49495f65a9bf6b449"
format_stamp: "Formatted at 2026-09-26 07:32:47 on dist-test-slave-1dwd"
server_key: "58f5551bb66ee32cb77dd713f60b5c45"
server_key_iv: "b8a0f04f644d33ed32997feddcf46064"
server_key_version: "encryptionkey@0"
I20260926 07:32:47.243916 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:47.247303 24473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:47.248564 24458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:32:47.248744 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "6bab28c52ad943f49495f65a9bf6b449"
format_stamp: "Formatted at 2026-09-26 07:32:47 on dist-test-slave-1dwd"
server_key: "58f5551bb66ee32cb77dd713f60b5c45"
server_key_iv: "b8a0f04f644d33ed32997feddcf46064"
server_key_version: "encryptionkey@0"
I20260926 07:32:47.248941 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:47.275969 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:47.279815 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:47.280145 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:47.298157 24458 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41187
I20260926 07:32:47.298180 24524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41187 every 8 connection(s)
I20260926 07:32:47.299489 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:47.303021 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24458
I20260926 07:32:47.303062 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:47.303256 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:47.303671 2639 external_mini_cluster.cc:1428] Setting key 72df7f319c44c9069d57fd39dc21766f
I20260926 07:32:47.316439 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: Bootstrap starting.
I20260926 07:32:47.316495 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.305342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58859 (local address 127.2.147.254:41187)
0926 07:32:47.306131 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:47.306142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:47.306183 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:47.307831 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:47.307856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:47.308225 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:47.308541 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:47.309119 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.310444 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.312225 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.312866 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.313011 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.314281 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:47.314290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:47.314303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:47.314840 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:47.314913 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:47.314943 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:47.315053 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:47.315409 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 07:32:47.315586 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":368,"threads_started":1}
I20260926 07:32:47.319990 24525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:47.320170 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:47.321033 24525 log.cc:824] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:47.323767 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: No bootstrap required, opened a new log
I20260926 07:32:47.331041 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.324635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43963 (local address 127.2.147.254:41187)
0926 07:32:47.324844 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:47.324850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:47.325299 (+ 449us) server_negotiation.cc:407] Connection header received
0926 07:32:47.325443 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:47.325453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:47.325533 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:47.325639 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:47.326090 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.327232 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.328459 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.329144 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.329271 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.330266 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:47.330272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:47.330277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:47.330298 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:47.330363 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:47.330388 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:47.330527 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:47.330656 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 07:32:47.330704 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:47.333024 24525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } }
I20260926 07:32:47.333395 24525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:47.333541 24525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bab28c52ad943f49495f65a9bf6b449, State: Initialized, Role: FOLLOWER
I20260926 07:32:47.334108 24525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } }
I20260926 07:32:47.334336 24525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:47.334496 24525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:47.334652 24525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:47.336855 24525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } }
I20260926 07:32:47.337349 24525 leader_election.cc:302] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bab28c52ad943f49495f65a9bf6b449; no voters:
I20260926 07:32:47.337999 24525 leader_election.cc:288] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:47.338112 24535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:47.340623 24525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:47.349515 24535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [term 1 LEADER]: Becoming Leader. State: Replica: 6bab28c52ad943f49495f65a9bf6b449, State: Running, Role: LEADER
I20260926 07:32:47.350313 24535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } }
I20260926 07:32:47.355937 24537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bab28c52ad943f49495f65a9bf6b449. Latest consensus state: current_term: 1 leader_uuid: "6bab28c52ad943f49495f65a9bf6b449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } } }
I20260926 07:32:47.356751 24537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:47.356456 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bab28c52ad943f49495f65a9bf6b449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bab28c52ad943f49495f65a9bf6b449" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41187 } } }
I20260926 07:32:47.357175 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449 [sys.catalog]: This master's current role is: LEADER
W20260926 07:32:47.360628 24549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:47.360803 24549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:47.360982 24550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:47.362565 24550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:47.370162 24550 catalog_manager.cc:1347] Generated new cluster ID: c3f279978ca944dda0022fbdc6c1fbf6
I20260926 07:32:47.370293 24550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:47.396651 24550 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:47.398557 24550 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:47.410727 24550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bab28c52ad943f49495f65a9bf6b449: Generated new TSK 0
I20260926 07:32:47.411900 24550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:47.429883 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24458
W20260926 07:32:47.430951 24520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58859
W20260926 07:32:47.431747 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:47.431869 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24458
W20260926 07:32:47.448372 24443 connection.cc:535] client connection to 127.2.147.254:41187 recv error: Network error: failed to read from TLS socket (remote: 127.2.147.254:41187): Connection reset by peer (error 104)
2026-09-26T07:32:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:47 dist-test-slave-1dwd krb5kdc[24556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:47 dist-test-slave-1dwd krb5kdc[24556](info): set up 2 sockets
Sep 26 07:32:47 dist-test-slave-1dwd krb5kdc[24556](info): commencing operation
krb5kdc: starting...
W20260926 07:32:48.527747 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:48 dist-test-slave-1dwd krb5kdc[24556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:48.776337 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46011
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:49.096931 24569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:49.097267 24569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:49.097448 24569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:49.106788 24569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:49.106952 24569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:49.107046 24569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:49.107151 24569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:49.107236 24569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:49.118594 24569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:49.119984 24569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:49.121307 24569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:49.133028 24574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:49.133061 24575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:49.132992 24577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:49.134135 24569 server_base.cc:1034] running on GCE node
I20260926 07:32:49.134922 24569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:49.136439 24569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:49.137686 24569 hybrid_clock.cc:648] HybridClock initialized: now 1790407969137613 us; error 50 us; skew 500 ppm
I20260926 07:32:49.138118 24569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:49 dist-test-slave-1dwd krb5kdc[24556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407969, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:49.141346 24569 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:49.142829 24569 webserver.cc:460] Webserver started at http://127.2.147.254:45951/ using document root <none> and password file <none>
I20260926 07:32:49.143568 24569 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:49.143702 24569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:49.144012 24569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:49.146965 24569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "cace11b9ef7d4ce7b9724f26d66d0d44"
format_stamp: "Formatted at 2026-09-26 07:32:49 on dist-test-slave-1dwd"
server_key: "60cce684b7c03d86aa80dbbba0c20b17"
server_key_iv: "81ae7405eedc93466e2c4b7fa156dd0b"
server_key_version: "encryptionkey@0"
I20260926 07:32:49.147773 24569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "cace11b9ef7d4ce7b9724f26d66d0d44"
format_stamp: "Formatted at 2026-09-26 07:32:49 on dist-test-slave-1dwd"
server_key: "60cce684b7c03d86aa80dbbba0c20b17"
server_key_iv: "81ae7405eedc93466e2c4b7fa156dd0b"
server_key_version: "encryptionkey@0"
I20260926 07:32:49.152783 24569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:32:49.156148 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:49.157315 24569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:49.157495 24569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "cace11b9ef7d4ce7b9724f26d66d0d44"
format_stamp: "Formatted at 2026-09-26 07:32:49 on dist-test-slave-1dwd"
server_key: "60cce684b7c03d86aa80dbbba0c20b17"
server_key_iv: "81ae7405eedc93466e2c4b7fa156dd0b"
server_key_version: "encryptionkey@0"
I20260926 07:32:49.157676 24569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:49.163177 24569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:49.166939 24569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:49.167290 24569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:49.185613 24569 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:46011
I20260926 07:32:49.185647 24635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:46011 every 8 connection(s)
I20260926 07:32:49.187027 24569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:49.188866 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24569
I20260926 07:32:49.189077 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:49.189553 2639 external_mini_cluster.cc:1428] Setting key 4ae6ccae9dea17ac80aaf1918ae8213d
I20260926 07:32:49.191289 24636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:49.198150 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:49.191303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33585 (local address 127.2.147.254:46011)
0926 07:32:49.192201 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:49.192216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:49.192267 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:32:49.194319 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:49.194343 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:49.194798 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:49.195265 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:49.196233 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:49.196242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:49.196258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:49.196287 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:49.196381 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:49.196405 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:49.196603 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:49.196739 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:32:49.196933 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":384,"threads_started":1}
I20260926 07:32:49.201774 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:49.204303 24636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44: Bootstrap starting.
I20260926 07:32:49.207873 24636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:49.208523 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:49.205922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36507 (local address 127.2.147.254:46011)
0926 07:32:49.206077 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:49.206082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:49.206551 (+ 469us) server_negotiation.cc:407] Connection header received
0926 07:32:49.206731 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:49.206736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:49.206794 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:49.206895 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:49.207871 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:49.207877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:49.207882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:49.207899 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:49.207981 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:49.208002 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:49.208110 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:49.208199 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:32:49.208260 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:49.209046 24636 log.cc:824] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:49.211730 24636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44: No bootstrap required, opened a new log
I20260926 07:32:49.218717 24636 raft_consensus.cc:348] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } }
I20260926 07:32:49.219106 24636 raft_consensus.cc:374] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:49.219231 24636 raft_consensus.cc:729] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cace11b9ef7d4ce7b9724f26d66d0d44, State: Initialized, Role: FOLLOWER
I20260926 07:32:49.219789 24636 consensus_queue.cc:260] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } }
I20260926 07:32:49.220031 24636 raft_consensus.cc:388] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:49.220173 24636 raft_consensus.cc:482] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:49.220340 24636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:49.222470 24636 raft_consensus.cc:504] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } }
I20260926 07:32:49.222939 24636 leader_election.cc:302] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cace11b9ef7d4ce7b9724f26d66d0d44; no voters:
I20260926 07:32:49.223639 24636 leader_election.cc:288] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:49.223762 24646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:49.224776 24646 raft_consensus.cc:686] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [term 1 LEADER]: Becoming Leader. State: Replica: cace11b9ef7d4ce7b9724f26d66d0d44, State: Running, Role: LEADER
I20260926 07:32:49.225327 24646 consensus_queue.cc:237] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } }
I20260926 07:32:49.226038 24636 sys_catalog.cc:564] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:49.228160 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } } }
I20260926 07:32:49.228497 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cace11b9ef7d4ce7b9724f26d66d0d44. Latest consensus state: current_term: 1 leader_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cace11b9ef7d4ce7b9724f26d66d0d44" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46011 } } }
I20260926 07:32:49.228708 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:49.228790 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:49.232601 24654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:49.237987 24654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:49.247195 24654 catalog_manager.cc:1347] Generated new cluster ID: 5d3f7ecba7674884b9252d3b44efce43
I20260926 07:32:49.247373 24654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:49.271982 24654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:49.272483 24654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:49.291642 24654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cace11b9ef7d4ce7b9724f26d66d0d44: Generated new TSK 0
I20260926 07:32:49.292634 24654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:49.309051 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24569
W20260926 07:32:49.310081 24631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33585
W20260926 07:32:49.310717 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:49.310871 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24569
2026-09-26T07:32:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:49 dist-test-slave-1dwd krb5kdc[24667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:49 dist-test-slave-1dwd krb5kdc[24667](info): set up 2 sockets
Sep 26 07:32:49 dist-test-slave-1dwd krb5kdc[24667](info): commencing operation
krb5kdc: starting...
W20260926 07:32:50.411674 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:50 dist-test-slave-1dwd krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:50.655485 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41609
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:50.974592 24680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:50.974850 24680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:50.974948 24680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:50.981863 24680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:50.981966 24680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:50.982012 24680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:50.982072 24680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:50.982115 24680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:50.991420 24680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:50.992525 24680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:50.993726 24680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:51.005324 24685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:51.005362 24686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:51.005416 24688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:51.006201 24680 server_base.cc:1034] running on GCE node
I20260926 07:32:51.006939 24680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:51.008379 24680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:51.009621 24680 hybrid_clock.cc:648] HybridClock initialized: now 1790407971009580 us; error 47 us; skew 500 ppm
I20260926 07:32:51.009972 24680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:51 dist-test-slave-1dwd krb5kdc[24667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407971, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:51.018141 24680 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:51.019908 24680 webserver.cc:460] Webserver started at http://127.2.147.254:39821/ using document root <none> and password file <none>
I20260926 07:32:51.020592 24680 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:51.020726 24680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:51.021037 24680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:51.024008 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "8c65d3f207714302b45798868de6ce79"
format_stamp: "Formatted at 2026-09-26 07:32:51 on dist-test-slave-1dwd"
server_key: "af4969e32fb5c04c71ace84991f7705d"
server_key_iv: "1e89068b5558effa10da868e7db95845"
server_key_version: "encryptionkey@0"
I20260926 07:32:51.024768 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "8c65d3f207714302b45798868de6ce79"
format_stamp: "Formatted at 2026-09-26 07:32:51 on dist-test-slave-1dwd"
server_key: "af4969e32fb5c04c71ace84991f7705d"
server_key_iv: "1e89068b5558effa10da868e7db95845"
server_key_version: "encryptionkey@0"
I20260926 07:32:51.029739 24680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:51.033046 24695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:51.034256 24680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:51.034438 24680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "8c65d3f207714302b45798868de6ce79"
format_stamp: "Formatted at 2026-09-26 07:32:51 on dist-test-slave-1dwd"
server_key: "af4969e32fb5c04c71ace84991f7705d"
server_key_iv: "1e89068b5558effa10da868e7db95845"
server_key_version: "encryptionkey@0"
I20260926 07:32:51.034623 24680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:51.057240 24680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:51.061101 24680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:51.061383 24680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:51.079447 24680 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:41609
I20260926 07:32:51.079506 24746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:41609 every 8 connection(s)
I20260926 07:32:51.080741 24680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:51.084184 24747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:51.089956 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24680
I20260926 07:32:51.090160 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:51.090674 2639 external_mini_cluster.cc:1428] Setting key 856343c9059fea665b86c263bbdd5a77
I20260926 07:32:51.095093 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79: Bootstrap starting.
I20260926 07:32:51.098666 24747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:51.099895 24747 log.cc:824] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:51.102907 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79: No bootstrap required, opened a new log
Sep 26 07:32:51 dist-test-slave-1dwd krb5kdc[24667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:51.114466 24747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } }
I20260926 07:32:51.114943 24747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:51.115094 24747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c65d3f207714302b45798868de6ce79, State: Initialized, Role: FOLLOWER
I20260926 07:32:51.115702 24747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } }
I20260926 07:32:51.115957 24747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:51.116115 24747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:51.116300 24747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:51.116463 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:51.092532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59393 (local address 127.2.147.254:41609)
0926 07:32:51.093377 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:51.093389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:51.093430 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:32:51.095153 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:51.095176 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:51.095624 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:51.095936 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:51.097664 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:51.099012 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:51.100796 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:51.101577 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:51.101705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:51.107177 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:51.107189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:51.107256 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:51.107336 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:51.111776 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:51.112590 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:51.112599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:51.112608 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:51.112864 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:51.113531 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:51.113535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:51.113537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:51.114042 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:51.114264 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:51.114708 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 07:32:51.115138 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":429,"threads_started":1}
I20260926 07:32:51.118811 24747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } }
I20260926 07:32:51.119439 24747 leader_election.cc:302] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c65d3f207714302b45798868de6ce79; no voters:
I20260926 07:32:51.120092 24747 leader_election.cc:288] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:51.120133 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:51.120208 24752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:51.122120 24747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:51.127842 24752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [term 1 LEADER]: Becoming Leader. State: Replica: 8c65d3f207714302b45798868de6ce79, State: Running, Role: LEADER
I20260926 07:32:51.128629 24752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } }
I20260926 07:32:51.131803 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c65d3f207714302b45798868de6ce79. Latest consensus state: current_term: 1 leader_uuid: "8c65d3f207714302b45798868de6ce79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } } }
I20260926 07:32:51.132190 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:51.132558 24753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c65d3f207714302b45798868de6ce79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c65d3f207714302b45798868de6ce79" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 41609 } } }
I20260926 07:32:51.132934 24753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:51.140316 24767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:51.146268 24767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:51.151192 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:51.130778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36725 (local address 127.2.147.254:41609)
0926 07:32:51.131370 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:51.131377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:51.132157 (+ 780us) server_negotiation.cc:407] Connection header received
0926 07:32:51.132370 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:51.132382 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:51.132489 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:51.132625 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:51.135086 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:51.136614 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:51.138958 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:51.140030 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:51.140198 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:51.143398 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:51.143408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:51.143417 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:51.143456 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:51.147426 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:51.148266 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:51.148285 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:51.148288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:51.148452 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:51.148890 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:51.148897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:51.148900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:51.149285 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:51.149513 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:51.150618 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0926 07:32:51.150817 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260926 07:32:51.155761 24767 catalog_manager.cc:1347] Generated new cluster ID: fad31cea4eb442c99e199620ce0c188c
I20260926 07:32:51.155862 24767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:51.168923 24767 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:51.170679 24767 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:51.189255 24767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c65d3f207714302b45798868de6ce79: Generated new TSK 0
I20260926 07:32:51.190296 24767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:51.198711 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24680
I20260926 07:32:51.199913 24742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:51.289726 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24680
2026-09-26T07:32:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:51 dist-test-slave-1dwd krb5kdc[24779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:51 dist-test-slave-1dwd krb5kdc[24779](info): set up 2 sockets
Sep 26 07:32:51 dist-test-slave-1dwd krb5kdc[24779](info): commencing operation
krb5kdc: starting...
W20260926 07:32:52.383708 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:52 dist-test-slave-1dwd krb5kdc[24779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:52.630899 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:35533
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:35533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:52.958913 24792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:52.959267 24792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:52.959439 24792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:52.966482 24792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:52.966641 24792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:52.966756 24792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:52.966861 24792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:52.966960 24792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:52.976593 24792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:35533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:35533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:52.977983 24792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:52.979360 24792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:52.990768 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:52.990722 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:52.990788 24800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:52.991577 24792 server_base.cc:1034] running on GCE node
I20260926 07:32:52.992331 24792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:52.993693 24792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:52.994930 24792 hybrid_clock.cc:648] HybridClock initialized: now 1790407972994891 us; error 44 us; skew 500 ppm
I20260926 07:32:52.995340 24792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:52 dist-test-slave-1dwd krb5kdc[24779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407972, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:52.998389 24792 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:52.999835 24792 webserver.cc:460] Webserver started at http://127.2.147.254:45365/ using document root <none> and password file <none>
I20260926 07:32:53.000496 24792 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:53.000645 24792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:53.001017 24792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:53.003973 24792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "e00c1248ad384004a1a4716a05e77307"
format_stamp: "Formatted at 2026-09-26 07:32:52 on dist-test-slave-1dwd"
server_key: "1fe69a6b70f3d0e8508c987477c5fb07"
server_key_iv: "79d658301fbecf9f000ed241a5b37ecb"
server_key_version: "encryptionkey@0"
I20260926 07:32:53.004761 24792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "e00c1248ad384004a1a4716a05e77307"
format_stamp: "Formatted at 2026-09-26 07:32:52 on dist-test-slave-1dwd"
server_key: "1fe69a6b70f3d0e8508c987477c5fb07"
server_key_iv: "79d658301fbecf9f000ed241a5b37ecb"
server_key_version: "encryptionkey@0"
I20260926 07:32:53.009828 24792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:32:53.013213 24807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:53.014544 24792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:53.014714 24792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "e00c1248ad384004a1a4716a05e77307"
format_stamp: "Formatted at 2026-09-26 07:32:52 on dist-test-slave-1dwd"
server_key: "1fe69a6b70f3d0e8508c987477c5fb07"
server_key_iv: "79d658301fbecf9f000ed241a5b37ecb"
server_key_version: "encryptionkey@0"
I20260926 07:32:53.014909 24792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:53.033332 24792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:53.037683 24792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:53.037964 24792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:53.056090 24792 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:35533
I20260926 07:32:53.056082 24858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:35533 every 8 connection(s)
I20260926 07:32:53.057420 24792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:53.060949 24859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:53.066325 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24792
I20260926 07:32:53.066514 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:53.066936 2639 external_mini_cluster.cc:1428] Setting key 35ccb0415ad9fac27aa6b25e5defd12d
I20260926 07:32:53.071467 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307: Bootstrap starting.
I20260926 07:32:53.074873 24859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:53.076053 24859 log.cc:824] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:53.079097 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307: No bootstrap required, opened a new log
I20260926 07:32:53.080030 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:53.068761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39471 (local address 127.2.147.254:35533)
0926 07:32:53.069605 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:53.069615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:53.069661 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:32:53.071313 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:53.071334 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:53.071754 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:53.072060 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:53.072606 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:53.073879 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:53.075732 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:53.076438 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:53.076662 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:53.077783 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:53.077790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:53.077803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:53.078325 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:53.078395 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:53.078437 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:53.078524 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:53.078761 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:32:53.079065 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":399,"threads_started":1}
I20260926 07:32:53.083477 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:53.086848 24859 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } }
I20260926 07:32:53.087278 24859 raft_consensus.cc:374] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:53.087409 24859 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00c1248ad384004a1a4716a05e77307, State: Initialized, Role: FOLLOWER
I20260926 07:32:53.088064 24859 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } }
I20260926 07:32:53.088361 24859 raft_consensus.cc:388] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:53.088537 24859 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:53.088734 24859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:53.090981 24859 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } }
I20260926 07:32:53.091511 24859 leader_election.cc:302] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e00c1248ad384004a1a4716a05e77307; no voters:
I20260926 07:32:53.092208 24859 leader_election.cc:288] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:53.092305 24869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:53.093792 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:53.087539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45661 (local address 127.2.147.254:35533)
0926 07:32:53.087700 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:53.087705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:53.088266 (+ 561us) server_negotiation.cc:407] Connection header received
0926 07:32:53.088457 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:53.088461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:53.088561 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:53.088667 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:53.089205 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:53.089939 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:53.091244 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:53.091761 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:53.091895 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:53.093075 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:53.093081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:53.093086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:53.093108 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:53.093175 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:53.093194 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:53.093248 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:53.093481 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:32:53.093564 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:32:53.094468 24869 raft_consensus.cc:686] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [term 1 LEADER]: Becoming Leader. State: Replica: e00c1248ad384004a1a4716a05e77307, State: Running, Role: LEADER
I20260926 07:32:53.094686 24859 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:53.095078 24869 consensus_queue.cc:237] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } }
I20260926 07:32:53.098269 24871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00c1248ad384004a1a4716a05e77307. Latest consensus state: current_term: 1 leader_uuid: "e00c1248ad384004a1a4716a05e77307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } } }
I20260926 07:32:53.098346 24870 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e00c1248ad384004a1a4716a05e77307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00c1248ad384004a1a4716a05e77307" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 35533 } } }
I20260926 07:32:53.098646 24871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:53.098692 24870 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:53.100668 24879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:53.110797 24879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:53.120807 24879 catalog_manager.cc:1347] Generated new cluster ID: bb924b97c1414d50b04a5f617110a389
I20260926 07:32:53.120949 24879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:53.145772 24879 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:53.147785 24879 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:53.157666 24879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e00c1248ad384004a1a4716a05e77307: Generated new TSK 0
I20260926 07:32:53.158691 24879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:53.171772 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24792
W20260926 07:32:53.173238 24854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39471
W20260926 07:32:53.173910 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:53.174055 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24792
2026-09-26T07:32:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:53 dist-test-slave-1dwd krb5kdc[24890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:53 dist-test-slave-1dwd krb5kdc[24890](info): set up 2 sockets
Sep 26 07:32:53 dist-test-slave-1dwd krb5kdc[24890](info): commencing operation
krb5kdc: starting...
W20260926 07:32:54.267815 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:54 dist-test-slave-1dwd krb5kdc[24890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:54.516577 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46323
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:54.839722 24903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:54.840003 24903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:54.840206 24903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:54.847229 24903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:54.847383 24903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:54.847492 24903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:54.847605 24903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:54.847692 24903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:54.857409 24903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:46323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:54.858686 24903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:54.860036 24903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:54.871819 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:54.871922 24908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:54.871922 24911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:54.872751 24903 server_base.cc:1034] running on GCE node
I20260926 07:32:54.873768 24903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:54.875349 24903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:54.876590 24903 hybrid_clock.cc:648] HybridClock initialized: now 1790407974876498 us; error 80 us; skew 500 ppm
I20260926 07:32:54.876979 24903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:54 dist-test-slave-1dwd krb5kdc[24890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:54.880205 24903 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:54.881716 24903 webserver.cc:460] Webserver started at http://127.2.147.254:35213/ using document root <none> and password file <none>
I20260926 07:32:54.882411 24903 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:54.882553 24903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:54.882869 24903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:54.885874 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130"
format_stamp: "Formatted at 2026-09-26 07:32:54 on dist-test-slave-1dwd"
server_key: "a351a1fe9022a23f92e3bab4ecadabe9"
server_key_iv: "f8a2bba8f2f860fd5684d5e6ffa6d19c"
server_key_version: "encryptionkey@0"
I20260926 07:32:54.886629 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130"
format_stamp: "Formatted at 2026-09-26 07:32:54 on dist-test-slave-1dwd"
server_key: "a351a1fe9022a23f92e3bab4ecadabe9"
server_key_iv: "f8a2bba8f2f860fd5684d5e6ffa6d19c"
server_key_version: "encryptionkey@0"
I20260926 07:32:54.891819 24903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 07:32:54.895311 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:54.896648 24903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:54.896838 24903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130"
format_stamp: "Formatted at 2026-09-26 07:32:54 on dist-test-slave-1dwd"
server_key: "a351a1fe9022a23f92e3bab4ecadabe9"
server_key_iv: "f8a2bba8f2f860fd5684d5e6ffa6d19c"
server_key_version: "encryptionkey@0"
I20260926 07:32:54.897054 24903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:54.902922 24903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:54.907377 24903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:54.907671 24903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:54.926018 24903 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:46323
I20260926 07:32:54.926071 24969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:46323 every 8 connection(s)
I20260926 07:32:54.927534 24903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:54.930353 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 24903
I20260926 07:32:54.930557 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:54.930992 2639 external_mini_cluster.cc:1428] Setting key 897b8bd4ba088815b8c9909ec68781c3
I20260926 07:32:54.931521 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:54.939668 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:54.932855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37317 (local address 127.2.147.254:46323)
0926 07:32:54.933627 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:54.933638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:54.933677 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:32:54.936146 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:54.936169 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:54.936523 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:54.936972 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:54.938064 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:54.938070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:54.938083 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:54.938098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:54.938150 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:54.938168 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:54.938409 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:54.938561 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:32:54.938726 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":340,"threads_started":1}
I20260926 07:32:54.943130 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:54.944612 24970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: Bootstrap starting.
I20260926 07:32:54.947950 24970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:54.949115 24970 log.cc:824] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:54.949954 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:54.947273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45651 (local address 127.2.147.254:46323)
0926 07:32:54.947452 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:54.947460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:54.947851 (+ 391us) server_negotiation.cc:407] Connection header received
0926 07:32:54.948038 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:54.948043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:54.948128 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:54.948237 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:54.949165 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:54.949171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:54.949176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:32:54.949192 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:54.949254 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:32:54.949275 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:54.949376 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:54.949503 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 07:32:54.949565 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:32:54.952008 24970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: No bootstrap required, opened a new log
I20260926 07:32:54.958817 24970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } }
I20260926 07:32:54.959170 24970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:54.959306 24970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f601d54ea9f4c4b9bd864c8d2ff9130, State: Initialized, Role: FOLLOWER
I20260926 07:32:54.959832 24970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } }
I20260926 07:32:54.960069 24970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:54.960206 24970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:54.960389 24970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:54.962447 24970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } }
I20260926 07:32:54.962927 24970 leader_election.cc:302] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f601d54ea9f4c4b9bd864c8d2ff9130; no voters:
I20260926 07:32:54.963567 24970 leader_election.cc:288] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:54.963692 24980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:54.964406 24980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [term 1 LEADER]: Becoming Leader. State: Replica: 5f601d54ea9f4c4b9bd864c8d2ff9130, State: Running, Role: LEADER
I20260926 07:32:54.964905 24980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } }
I20260926 07:32:54.965624 24970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:54.972136 24982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f601d54ea9f4c4b9bd864c8d2ff9130. Latest consensus state: current_term: 1 leader_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } } }
I20260926 07:32:54.972597 24982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:54.972177 24981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f601d54ea9f4c4b9bd864c8d2ff9130" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 46323 } } }
I20260926 07:32:54.973220 24981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130 [sys.catalog]: This master's current role is: LEADER
W20260926 07:32:54.976869 24994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:54.977025 24994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:54.981166 24995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:54.982779 24995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:54.990515 24995 catalog_manager.cc:1347] Generated new cluster ID: da1e989dafaf4eac8f1f0bddd49b251d
I20260926 07:32:54.990634 24995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:55.004021 24995 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:55.004529 24995 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:55.013883 24995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f601d54ea9f4c4b9bd864c8d2ff9130: Generated new TSK 0
I20260926 07:32:55.015018 24995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:55.031602 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24903
W20260926 07:32:55.032686 24965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37317
W20260926 07:32:55.033380 2639 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 24903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 07:32:55.033530 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 24903
2026-09-26T07:32:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1859 ms)
[----------] 18 tests from AuthTokenIssuingTest (34549 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:55 dist-test-slave-1dwd krb5kdc[25001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:55 dist-test-slave-1dwd krb5kdc[25001](info): set up 2 sockets
Sep 26 07:32:55 dist-test-slave-1dwd krb5kdc[25001](info): commencing operation
krb5kdc: starting...
W20260926 07:32:56.143913 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:32:56 dist-test-slave-1dwd krb5kdc[25001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:32:56.399164 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:33643
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:56.724876 25014 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:32:56.725173 25014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:56.725234 25014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:56.725304 25014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:56.732365 25014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:56.732452 25014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:56.732508 25014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:56.732558 25014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:56.732611 25014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:56.741894 25014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:33643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:56.743052 25014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:56.744342 25014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:56.758060 25019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:56.758023 25020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:56.758348 25022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:56.759184 25014 server_base.cc:1034] running on GCE node
I20260926 07:32:56.759939 25014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:56.761211 25014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:56.762420 25014 hybrid_clock.cc:648] HybridClock initialized: now 1790407976762363 us; error 41 us; skew 500 ppm
I20260926 07:32:56.762745 25014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:56 dist-test-slave-1dwd krb5kdc[25001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:56.771426 25014 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:32:56.772882 25014 webserver.cc:460] Webserver started at http://127.2.147.254:35911/ using document root <none> and password file <none>
I20260926 07:32:56.773572 25014 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:56.773705 25014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:56.774029 25014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:56.776949 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "3df14706b4f4474e89305b9d55b4654b"
format_stamp: "Formatted at 2026-09-26 07:32:56 on dist-test-slave-1dwd"
server_key: "822153ec1f3406a7a0be9c4f4e85b321"
server_key_iv: "193a28815dddf80a90b1905a51392391"
server_key_version: "encryptionkey@0"
I20260926 07:32:56.777707 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "3df14706b4f4474e89305b9d55b4654b"
format_stamp: "Formatted at 2026-09-26 07:32:56 on dist-test-slave-1dwd"
server_key: "822153ec1f3406a7a0be9c4f4e85b321"
server_key_iv: "193a28815dddf80a90b1905a51392391"
server_key_version: "encryptionkey@0"
I20260926 07:32:56.782627 25014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 07:32:56.785916 25029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:56.787132 25014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:32:56.787348 25014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "3df14706b4f4474e89305b9d55b4654b"
format_stamp: "Formatted at 2026-09-26 07:32:56 on dist-test-slave-1dwd"
server_key: "822153ec1f3406a7a0be9c4f4e85b321"
server_key_iv: "193a28815dddf80a90b1905a51392391"
server_key_version: "encryptionkey@0"
I20260926 07:32:56.787564 25014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:56.809392 25014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:56.814018 25014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:56.814304 25014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:56.831293 25014 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:33643
I20260926 07:32:56.831314 25080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:33643 every 8 connection(s)
I20260926 07:32:56.832603 25014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:56.834968 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25014
I20260926 07:32:56.835179 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:32:56.835983 2639 external_mini_cluster.cc:1428] Setting key a80b79c6351e2c8d8a94b66564af990b
I20260926 07:32:56.836781 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:56.846480 25081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:56 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.253@KRBTEST.COM" created.
I20260926 07:32:56.870936 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.849645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.2.147.254:33643)
0926 07:32:56.851335 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:56.851345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:56.851394 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:32:56.853044 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:56.853054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:56.853371 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:56.853592 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:56.854701 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:56.856715 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:56.858061 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:56.859073 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:56.859297 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:56.863106 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:56.863118 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:56.863133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:56.863176 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:56.866339 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:56.867112 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:56.867122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:56.867129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:56.867344 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:56.867805 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:56.867808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:56.867810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:56.868299 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:56.868456 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:56.869189 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 07:32:56.869525 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"spinlock_wait_cycles":51968,"thread_start_us":1215,"threads_started":1}
I20260926 07:32:56.870973 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.849646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:33643 (local address 127.0.0.1:34430)
0926 07:32:56.850639 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.850714 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:56.852657 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:56.853608 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:56.853613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:56.854153 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:56.854493 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:56.854509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:56.856870 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:56.856874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:56.857915 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:56.857926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:56.859344 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:56.859349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:56.859476 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:56.860185 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:56.860222 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:56.862859 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:56.866464 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:56.866473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:56.866482 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:56.866991 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:56.867463 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:56.867467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:56.867470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:56.867698 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:56.868455 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:56.868466 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:56.868744 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 07:32:56.869050 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 07:32:56.869757 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":148,"thread_start_us":378,"threads_started":1}
I20260926 07:32:56.873824 25081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:56.874817 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.874463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56542)
0926 07:32:56.874613 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.874714 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:32:56.875164 25030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
I20260926 07:32:56.876731 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.876338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56546)
0926 07:32:56.876470 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.876570 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:32:56.877044 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:37605: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
I20260926 07:32:56.899075 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:37605
--webserver_interface=127.2.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:56.926399 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } attempt: 1
I20260926 07:32:56.927647 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.927192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56550)
0926 07:32:56.927400 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.927513 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:32:56.928682 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.928304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56558)
0926 07:32:56.928478 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.928550 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 07:32:56.928983 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:37605: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
I20260926 07:32:56.997236 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } attempt: 2
I20260926 07:32:56.998457 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.997979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56574)
0926 07:32:56.998236 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.998327 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:32:56.999573 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.999199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56578)
0926 07:32:56.999370 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.999462 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 07:32:56.999802 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:37605: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
I20260926 07:32:57.091074 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } attempt: 3
I20260926 07:32:57.092332 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.091893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56590)
0926 07:32:57.092101 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.092192 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 07:32:57.093369 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.093047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56598)
0926 07:32:57.093174 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.093252 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:32:57.093588 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:37605: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
W20260926 07:32:57.223629 25088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:32:57.223937 25088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:57.224009 25088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:57.224089 25088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:57.231060 25088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:57.231145 25088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:57.231218 25088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:57.231263 25088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:57.231316 25088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:57.236828 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } attempt: 4
I20260926 07:32:57.238013 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.237507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56600)
0926 07:32:57.237788 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.237882 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260926 07:32:57.238957 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.238622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56602)
0926 07:32:57.238761 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.238845 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:32:57.239153 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:37605: Network error: Client connection negotiation failed: client connection to 127.2.147.253:37605: connect: Connection refused (error 111)
I20260926 07:32:57.241277 25088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:57.242540 25088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:57.243922 25088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:57.255512 25093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:57.255466 25096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:57.255561 25094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:57.260516 25088 server_base.cc:1034] running on GCE node
I20260926 07:32:57.261234 25088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:57.262555 25088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:57.263793 25088 hybrid_clock.cc:648] HybridClock initialized: now 1790407977263744 us; error 47 us; skew 500 ppm
I20260926 07:32:57.264117 25088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:57.267262 25088 init.cc:377] Logged in from keytab as kudu/127.2.147.253@KRBTEST.COM (short username kudu)
I20260926 07:32:57.268638 25088 webserver.cc:460] Webserver started at http://127.2.147.253:41217/ using document root <none> and password file <none>
I20260926 07:32:57.269330 25088 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:57.269462 25088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:57.269805 25088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:57.272951 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data/instance:
uuid: "599f0723f7ae479f9d9a1728f67cfede"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "12c9d6da96b8d38da778d9b49e8a6d8d"
server_key_iv: "b574e8b9ddafd72b8d48dbaeb7b083a4"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.273717 25088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal/instance:
uuid: "599f0723f7ae479f9d9a1728f67cfede"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "12c9d6da96b8d38da778d9b49e8a6d8d"
server_key_iv: "b574e8b9ddafd72b8d48dbaeb7b083a4"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.278705 25088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:32:57.282042 25103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:57.283411 25088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:32:57.283578 25088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "599f0723f7ae479f9d9a1728f67cfede"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "12c9d6da96b8d38da778d9b49e8a6d8d"
server_key_iv: "b574e8b9ddafd72b8d48dbaeb7b083a4"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.283804 25088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:57.295949 25088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:57.300272 25088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:57.300587 25088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:57.318786 25088 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.253:37605
I20260926 07:32:57.318794 25154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.253:37605 every 8 connection(s)
I20260926 07:32:57.320235 25088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
I20260926 07:32:57.323168 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25088
I20260926 07:32:57.323372 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-1/wal/instance
I20260926 07:32:57.323890 2639 external_mini_cluster.cc:1428] Setting key 38e3fcf0bc92f9a78d52f39eb4a047a7
I20260926 07:32:57.324311 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:57.334343 25155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.252@KRBTEST.COM" created.
I20260926 07:32:57.355551 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.337052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.2.147.254:33643)
0926 07:32:57.337290 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.337298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.338441 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 07:32:57.340344 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.340352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.340460 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.340593 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.341957 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.343123 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.344673 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.345473 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.345617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.349045 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.349054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.349060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.349106 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.352319 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.353037 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.353041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.353043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.353216 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.353755 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.353759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.353761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.354032 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.354144 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.355099 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.355322 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:57.357122 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.337458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:33643 (local address 127.0.0.1:34434)
0926 07:32:57.338247 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.338290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.340071 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.340768 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.340776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.341253 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.341790 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.341806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.343319 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.343323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.344510 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.344521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.345666 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.345670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.345865 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.346616 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.346637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.348797 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.352444 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.352451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.352463 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.352930 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.353385 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.353397 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.353400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.353629 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.354172 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.354183 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.354692 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.355027 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.355632 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":343,"threads_started":1}
I20260926 07:32:57.358938 25155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:32:57.373143 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.359646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56606)
0926 07:32:57.359803 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.359818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.359896 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.360854 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.360858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.361293 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.361416 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.361424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.362727 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.362733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.363671 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.363682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.364503 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.364509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.364676 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.365282 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.365299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.367502 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.370633 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.370640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.370646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.370990 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.371425 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.371431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.371434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.371571 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.372161 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.372168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.372319 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.372459 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.372722 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:32:57.373203 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.359905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.2.147.253:37605)
0926 07:32:57.360433 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.360439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.360479 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:57.360517 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.360523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.360710 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.360867 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.361588 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.362596 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.363810 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.364327 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.364485 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.367708 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.367718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.367730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.367779 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.370512 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.371149 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.371156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.371158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.371314 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.371679 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.371698 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.371703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.372040 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.372249 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.372613 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.372808 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260926 07:32:57.375581 25155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:57.376922 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.376379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43190)
0926 07:32:57.376582 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.376728 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 07:32:57.377241 25106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.378860 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.378443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43194)
0926 07:32:57.378642 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.378751 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:32:57.379077 25155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
I20260926 07:32:57.387038 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:38211
--webserver_interface=127.2.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:57.428452 25155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 1
I20260926 07:32:57.429768 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.429199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43204)
0926 07:32:57.429446 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.429574 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:32:57.430845 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.430423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43218)
0926 07:32:57.430616 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.430707 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:32:57.431079 25155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.499321 25155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 2
I20260926 07:32:57.500545 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.499999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43228)
0926 07:32:57.500239 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.500358 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:32:57.501549 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.501166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43240)
0926 07:32:57.501317 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.501407 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:32:57.501807 25155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.538455 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } attempt: 5
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:32:57.551970 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.539440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.2.147.253:37605)
0926 07:32:57.539647 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.539653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.539675 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:32:57.539777 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.539782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.539870 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.539981 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.540752 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.541460 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.542602 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.543055 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.543191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.546716 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.546723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.546727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.546769 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.549387 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.550028 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.550036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.550039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.550166 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.550555 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.550560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.550564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.550859 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.551043 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.551543 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.551734 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:32:57.552239 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.539163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56608)
0926 07:32:57.539482 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.539527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.539659 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.540019 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.540024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.540434 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.540604 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.540613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.541592 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.541597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.542457 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.542468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.543231 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.543238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.543450 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.544157 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.544174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.546524 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.549530 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.549535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.549539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.549907 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.550281 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.550286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.550290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.550436 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.551040 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.551046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.551271 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.551442 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.551787 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260926 07:32:57.553388 25081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:57.554553 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.554109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43248)
0926 07:32:57.554291 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.554394 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:32:57.555603 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.555166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43252)
0926 07:32:57.555342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.555462 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 07:32:57.555831 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.593040 25155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 3
I20260926 07:32:57.594432 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.593780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43266)
0926 07:32:57.594058 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.594187 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 07:32:57.595590 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.595094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43280)
0926 07:32:57.595370 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.595465 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260926 07:32:57.595813 25155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.607079 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 1
I20260926 07:32:57.608260 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.607807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43286)
0926 07:32:57.608004 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.608100 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:32:57.609252 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.608899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43302)
0926 07:32:57.609046 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.609135 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:32:57.609606 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.677852 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 2
I20260926 07:32:57.679044 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.678574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43318)
0926 07:32:57.678796 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.678884 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:32:57.680040 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.679720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43334)
0926 07:32:57.679867 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.679946 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:32:57.680243 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
W20260926 07:32:57.717483 25162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:32:57.717825 25162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:57.717947 25162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:57.718129 25162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:57.725315 25162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:57.725399 25162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:57.725474 25162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:57.725543 25162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:57.725654 25162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:57.735404 25162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:33643,127.2.147.253:37605,127.2.147.252:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:32:57.736690 25162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:57.738005 25162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:32:57.739040 25155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 4
I20260926 07:32:57.740236 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.739698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43348)
0926 07:32:57.739949 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.740056 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:32:57.741283 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.740902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43360)
0926 07:32:57.741070 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.741157 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:32:57.741564 25155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
W20260926 07:32:57.749679 25167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:57.749775 25168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:57.749795 25170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:57.750694 25162 server_base.cc:1034] running on GCE node
I20260926 07:32:57.751451 25162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:57.752759 25162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:57.754002 25162 hybrid_clock.cc:648] HybridClock initialized: now 1790407977753945 us; error 36 us; skew 500 ppm
I20260926 07:32:57.754318 25162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:57.757386 25162 init.cc:377] Logged in from keytab as kudu/127.2.147.252@KRBTEST.COM (short username kudu)
I20260926 07:32:57.758711 25162 webserver.cc:460] Webserver started at http://127.2.147.252:40389/ using document root <none> and password file <none>
I20260926 07:32:57.759385 25162 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:57.759472 25162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:57.759721 25162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:57.762478 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data/instance:
uuid: "04823201883340ed9654d2292677fc2c"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "d5db019162c998ee00f6e0c23451f954"
server_key_iv: "54dd3040d456caa239cfe37a0556b6ea"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.763154 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal/instance:
uuid: "04823201883340ed9654d2292677fc2c"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "d5db019162c998ee00f6e0c23451f954"
server_key_iv: "54dd3040d456caa239cfe37a0556b6ea"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.767918 25162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:57.770998 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:57.772284 25162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:32:57.772441 25162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "04823201883340ed9654d2292677fc2c"
format_stamp: "Formatted at 2026-09-26 07:32:57 on dist-test-slave-1dwd"
server_key: "d5db019162c998ee00f6e0c23451f954"
server_key_iv: "54dd3040d456caa239cfe37a0556b6ea"
server_key_version: "encryptionkey@0"
I20260926 07:32:57.772646 25162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:57.786528 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 3
I20260926 07:32:57.787632 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.787115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43372)
0926 07:32:57.787314 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.787460 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:32:57.788659 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.788267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43376)
0926 07:32:57.788473 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.788567 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 07:32:57.788892 25081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:38211: Network error: Client connection negotiation failed: client connection to 127.2.147.252:38211: connect: Connection refused (error 111)
I20260926 07:32:57.793155 25162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:57.797298 25162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:57.797570 25162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:57.815804 25162 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.252:38211
I20260926 07:32:57.815827 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.252:38211 every 8 connection(s)
I20260926 07:32:57.817104 25162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
I20260926 07:32:57.820688 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:57.821659 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25162
I20260926 07:32:57.821841 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407623164223-2639-0/minicluster-data/master-2/wal/instance
I20260926 07:32:57.822263 2639 external_mini_cluster.cc:1428] Setting key fff12bbb48e3b2c42adccae81e7bd37e
I20260926 07:32:57.830217 25229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:57.846460 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.824145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.2.147.252:38211)
0926 07:32:57.824905 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.824926 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.824977 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:32:57.827124 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.827151 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.827669 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.828108 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.829354 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.831335 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.833184 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.834196 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.834363 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.839102 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.839113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.839130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.839176 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.842612 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.843416 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.843423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.843428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.843566 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.843960 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.843964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.843966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.844414 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.844696 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.845139 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.845507 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":295,"threads_started":1}
I20260926 07:32:57.847585 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.832865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.2.147.254:33643)
0926 07:32:57.833093 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.833102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.833802 (+ 700us) server_negotiation.cc:407] Connection header received
0926 07:32:57.834080 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.834084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.834144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.834246 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.835657 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.836728 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.838224 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.838648 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.838776 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.842307 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.842312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.842315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.842346 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.844860 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.845573 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.845580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.845583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.845717 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.846089 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.846095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.846098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.846340 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.846511 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.846991 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.847232 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:32:57.847661 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.832835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:33643 (local address 127.0.0.1:34436)
0926 07:32:57.833667 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.833713 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.833927 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.834306 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.834313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.835107 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.835494 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.835508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.836895 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.836900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.838072 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.838084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.838866 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.838875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.839059 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.839726 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.839747 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.842119 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.845005 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.845017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.845023 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.845479 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.845835 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.845845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.845849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.845978 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.846492 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.846499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.846681 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.846888 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.847232 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":468,"threads_started":1}
I20260926 07:32:57.848897 25229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:32:57.862897 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.849547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:37605 (local address 127.0.0.1:56622)
0926 07:32:57.849955 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.849971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.850069 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.850440 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.850447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.850826 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.850961 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.850969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.852083 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.852088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.853058 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.853069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.853909 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.853915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.854103 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.854675 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.854700 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.857033 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.860323 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.860328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.860333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.860824 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.861168 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.861176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.861180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.861343 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.861964 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.861967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.862126 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.862293 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.862554 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 07:32:57.862982 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.849653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.2.147.253:37605)
0926 07:32:57.849833 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.849839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.850055 (+ 216us) server_negotiation.cc:407] Connection header received
0926 07:32:57.850175 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.850179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.850270 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.850405 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.851106 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.851948 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.853210 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.853781 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.853872 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.857228 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.857235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.857239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.857281 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.860204 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.860937 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.860941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.860943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.861063 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.861479 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.861483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.861487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.861831 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.861998 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.862434 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.862686 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:32:57.864003 25229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:57.864319 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.849653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.2.147.254:33643)
0926 07:32:57.849796 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.849801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.849929 (+ 128us) server_negotiation.cc:407] Connection header received
0926 07:32:57.850061 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.850065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.850137 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.850219 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.851478 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.852299 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.853589 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.854056 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.854174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.858331 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.858335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.858338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.858367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.861324 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.862122 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.862126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.862129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.862284 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.862732 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.862738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.862741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.863091 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.863363 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.863798 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.864041 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:32:57.877146 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.865059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.2.147.252:38211)
0926 07:32:57.865237 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.865243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.865261 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:57.865302 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.865306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.865365 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.865455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.866240 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.867347 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.868434 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.868802 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.868930 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.871845 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.871850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.871854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.871879 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.874522 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.875123 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.875130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.875135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.875298 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.875765 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.875769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.875771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.876120 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.876264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.876711 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.876875 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":59}
I20260926 07:32:57.877493 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.864804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43390)
0926 07:32:57.864981 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.864999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.865087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.865484 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.865487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.865949 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.866110 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.866118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.867461 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.867464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.868307 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.868316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.868938 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.868943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.869211 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.869781 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.869796 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.871659 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.874646 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.874654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.874658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.875014 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.875436 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.875439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.875442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.875649 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.876263 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.876268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.876424 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.876621 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.877092 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":123}
I20260926 07:32:57.880592 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.865936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.2.147.253:37605)
0926 07:32:57.866084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.866090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.866106 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:32:57.866287 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.866291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.866341 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.866416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.867832 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.868492 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.869962 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.870373 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.870511 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.874832 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.874838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.874849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.874880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.877977 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.878608 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.878614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.878618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.878756 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.879154 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.879159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.879163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.879487 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.879796 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.880069 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.880299 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:57.882849 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: Bootstrap starting.
I20260926 07:32:57.885217 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:57.886204 25229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:57.887197 25229 log.cc:824] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:57.889822 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: No bootstrap required, opened a new log
I20260926 07:32:57.888038 25056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3df14706b4f4474e89305b9d55b4654b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34448
I20260926 07:32:57.897033 25229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:57.897357 25229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04823201883340ed9654d2292677fc2c, State: Initialized, Role: FOLLOWER
I20260926 07:32:57.897778 25229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:57.898692 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:57.899052 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:57.899806 25229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:32:57.909024 25246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:57.909157 25246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:57.906622 25162 master_runner.cc:186] Error getting master registration for 127.2.147.254:33643: OK, instance_id { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" instance_seqno: 1790407976826427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:32:57.914875 25162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:32:57.966135 25081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 4
Sep 26 07:32:57 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:32:57.981938 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.967138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.2.147.252:38211)
0926 07:32:57.967379 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:57.967384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:57.967405 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:32:57.967519 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:57.967525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:57.967628 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:57.967734 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:57.968739 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.969915 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.971712 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.972176 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.972302 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.976271 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:57.976277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:57.976282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:57.976321 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:57.979395 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.980157 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.980160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.980162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.980276 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:57.980659 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:57.980662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:57.980664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:57.980977 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:57.981106 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:57.981520 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 07:32:57.981678 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:32:57.982069 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.966956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43398)
0926 07:32:57.967207 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.967258 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:57.967392 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:57.967824 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:57.967830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:57.968334 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:57.968554 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.968584 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.970049 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.970055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.971569 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:57.971584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:57.972307 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:57.972313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:57.972523 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:57.973212 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:57.973237 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:57.976075 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:57.979512 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.979517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.979522 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.980040 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.980369 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:57.980374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:57.980382 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:57.980546 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:57.981085 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:57.981090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:57.981256 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 07:32:57.981412 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 07:32:57.981690 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:32:57.988152 25081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: Bootstrap starting.
I20260926 07:32:57.992272 25081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:57.993558 25081 log.cc:824] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:57.996863 25081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: No bootstrap required, opened a new log
I20260926 07:32:58.004220 25081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.004623 25081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df14706b4f4474e89305b9d55b4654b, State: Initialized, Role: FOLLOWER
I20260926 07:32:58.005091 25081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.006107 25249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.006469 25249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.007325 25081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:58.007424 25056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:32:58.007642 25056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:58.010605 25056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.012797 25204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3df14706b4f4474e89305b9d55b4654b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04823201883340ed9654d2292677fc2c"
I20260926 07:32:58.013273 25204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:58.016680 25204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df14706b4f4474e89305b9d55b4654b in term 1.
I20260926 07:32:58.017180 25056 leader_election.cc:288] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [CANDIDATE]: Term 1 election: Requested vote from peers 599f0723f7ae479f9d9a1728f67cfede (127.2.147.253:37605), 04823201883340ed9654d2292677fc2c (127.2.147.252:38211)
I20260926 07:32:58.017302 25032 leader_election.cc:302] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04823201883340ed9654d2292677fc2c, 3df14706b4f4474e89305b9d55b4654b; no voters:
I20260926 07:32:58.017874 25249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:58.019382 25249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [term 1 LEADER]: Becoming Leader. State: Replica: 3df14706b4f4474e89305b9d55b4654b, State: Running, Role: LEADER
I20260926 07:32:58.019972 25249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.025008 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df14706b4f4474e89305b9d55b4654b. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.026021 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:58.035910 25259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:58.040766 25155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } attempt: 5
W20260926 07:32:58.049504 25183 tablet.cc:2369] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:32:58 dist-test-slave-1dwd krb5kdc[25001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407977, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:32:58.050561 25259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:58.058034 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.041414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:38211 (local address 127.0.0.1:43410)
0926 07:32:58.041805 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:58.041826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:58.041944 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:58.042787 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:58.042793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:58.043331 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:58.043543 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:58.043554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.045023 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.045029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:58.046456 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:58.046472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.047625 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.047631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:58.047863 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.048569 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:58.048591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:58.051335 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:58.055285 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:58.055290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:58.055294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:58.055811 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:58.056235 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:58.056240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:58.056244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:58.056413 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:58.057056 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:58.057061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:58.057208 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 07:32:58.057368 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:32:58.057646 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260926 07:32:58.058939 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.042150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.2.147.252:38211)
0926 07:32:58.042317 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.042323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.042341 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:58.042413 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.042419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.042499 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.042591 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.043727 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.044852 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.046620 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.047317 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.047443 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.051549 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:58.051556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:58.051561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:58.051600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:58.055122 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.055954 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.055959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.055963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.056109 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.056539 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.056544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.056547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.056855 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:58.056996 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.058393 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.058599 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:32:58.064973 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: Bootstrap starting.
W20260926 07:32:58.067094 25035 tablet.cc:2369] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:58.069851 25155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:58.071254 25155 log.cc:824] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:58.081110 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: No bootstrap required, opened a new log
I20260926 07:32:58.085650 25204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [term 1 FOLLOWER]: Refusing update from remote peer 3df14706b4f4474e89305b9d55b4654b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:58.086715 25255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [LEADER]: Connected to new peer: Peer: permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:58.102227 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df14706b4f4474e89305b9d55b4654b. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.102579 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.102891 25265 mvcc.cc:205] Tried to move back new op lower bound from 7333511078227959808 to 7333511077975736320. Current Snapshot: MvccSnapshot[applied={T|T < 7333511078227959808}]
I20260926 07:32:58.115768 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.063375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.2.147.254:33643)
0926 07:32:58.063553 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.063559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.069594 (+ 6035us) server_negotiation.cc:407] Connection header received
0926 07:32:58.070001 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.070007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.070090 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.070176 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.073936 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.075112 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.078252 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.078961 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.079065 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.084821 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:58.084829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:58.084835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:58.084872 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:58.090059 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.112622 (+ 22563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.112628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.112632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.112830 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.113289 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.113294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.113297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.113659 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:58.113920 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.114965 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.115251 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:58.118485 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.118803 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:58.121444 25259 catalog_manager.cc:1347] Generated new cluster ID: 183365f56710468da20be30d62abd928
I20260926 07:32:58.121552 25259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:58.128993 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.067949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.2.147.252:38211)
0926 07:32:58.068104 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.068109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.077765 (+ 9656us) server_negotiation.cc:407] Connection header received
0926 07:32:58.078677 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.078685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.078777 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.078892 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.083844 (+ 4952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.084983 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.086730 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.087340 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.087445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.106799 (+ 19354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:58.106810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:58.106817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:58.106874 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:58.111868 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.120690 (+ 8822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.120697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.120700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.121037 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.123335 (+ 2298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.123341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.123344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.124147 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:58.124646 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.127619 (+ 2973us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.128177 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:32:58.132176 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.078242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.2.147.253:37605)
0926 07:32:58.078421 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.078426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.079992 (+ 1566us) server_negotiation.cc:407] Connection header received
0926 07:32:58.083421 (+ 3429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.083426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.083497 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.083564 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.104222 (+ 20658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.105300 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.109573 (+ 4273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.110249 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.110497 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.121414 (+ 10917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:58.121424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:58.121431 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:58.121478 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:58.125075 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.126459 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.126464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.126468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.126596 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:58.127284 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:58.127294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:58.127297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:58.127669 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:58.127872 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.131542 (+ 3670us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.131808 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:58.134205 25277 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:32:58.134533 25272 mvcc.cc:205] Tried to move back new op lower bound from 7333511078227959808 to 7333511077975736320. Current Snapshot: MvccSnapshot[applied={T|T < 7333511078227959808}]
I20260926 07:32:58.137777 25155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.138218 25155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599f0723f7ae479f9d9a1728f67cfede, State: Initialized, Role: FOLLOWER
I20260926 07:32:58.138744 25155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } }
I20260926 07:32:58.139681 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.139942 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.147029 25259 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:58.148977 25259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:58.174619 25259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b: Generated new TSK 0
I20260926 07:32:58.175889 25259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:58.137996 25014 master_runner.cc:186] Error getting master registration for 127.2.147.253:37605: OK, instance_id { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" instance_seqno: 1790407977313582 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:32:58.178587 25014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:32:58.188561 25278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.190008 25155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:58.190122 25278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.190466 25129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:58.194423 25129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [term 1 FOLLOWER]: Refusing update from remote peer 3df14706b4f4474e89305b9d55b4654b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:58.195449 25249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [LEADER]: Connected to new peer: Peer: permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:58.199330 25130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3df14706b4f4474e89305b9d55b4654b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "599f0723f7ae479f9d9a1728f67cfede"
I20260926 07:32:58.246348 25278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df14706b4f4474e89305b9d55b4654b. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.246696 25278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.247154 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.247519 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:58.247988 25130 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3df14706b4f4474e89305b9d55b4654b for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260926 07:32:58.273545 25295 catalog_manager.cc:1259] Loaded cluster ID: 183365f56710468da20be30d62abd928
I20260926 07:32:58.273748 25295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: loading cluster ID for follower catalog manager: success
I20260926 07:32:58.277786 25295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: acquiring CA information for follower catalog manager: success
I20260926 07:32:58.280429 25295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 599f0723f7ae479f9d9a1728f67cfede: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:32:58.304804 2639 mini_kdc.cc:348] Time spent kdestroy: real 0.105s user 0.000s sys 0.024s
W20260926 07:32:58.362115 25085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.340880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.2.147.254:33643)
0926 07:32:58.341070 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.341076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.359804 (+ 18728us) server_negotiation.cc:407] Connection header received
0926 07:32:58.360212 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.360218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.360302 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.360384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.361859 (+ 1475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34464: BlockingRecv error: recv got EOF from 127.0.0.1:34464 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 07:32:58.362264 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.347874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.2.147.252:38211)
0926 07:32:58.348062 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.348068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.360117 (+ 12049us) server_negotiation.cc:407] Connection header received
0926 07:32:58.360563 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.360570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.360648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.360739 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.362013 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43436: BlockingRecv error: recv got EOF from 127.0.0.1:43436 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:32:58.364405 25159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.360694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.2.147.253:37605)
0926 07:32:58.360898 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.360903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.362944 (+ 2041us) server_negotiation.cc:407] Connection header received
0926 07:32:58.363165 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.363171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.363288 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.363358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.364186 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56642: BlockingRecv error: recv got EOF from 127.0.0.1:56642 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:32:58.407140 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.396766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.2.147.253:37605)
0926 07:32:58.396978 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.396984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.399725 (+ 2741us) server_negotiation.cc:407] Connection header received
0926 07:32:58.400073 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.400080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.400272 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.400406 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.401114 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.402222 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.404244 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.404869 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.404965 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.405652 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.406324 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.406441 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.406693 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.406795 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:32:58.449090 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.439330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.2.147.252:38211)
0926 07:32:58.439530 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.439537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.446926 (+ 7389us) server_negotiation.cc:407] Connection header received
0926 07:32:58.447465 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.447472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.447636 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.447722 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.448803 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37810: BlockingRecv error: recv got EOF from 127.0.0.1:37810 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:58.498296 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.480168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.2.147.253:37605)
0926 07:32:58.480346 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.480352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.491171 (+ 10819us) server_negotiation.cc:407] Connection header received
0926 07:32:58.491817 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.491824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.491926 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.492001 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.492692 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.493895 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.495977 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.496689 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.496787 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.497410 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.497564 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.497643 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.497896 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.497993 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:32:58.543835 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.534353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.2.147.252:38211)
0926 07:32:58.534585 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.534592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.541902 (+ 7310us) server_negotiation.cc:407] Connection header received
0926 07:32:58.542370 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.542378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.542471 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.542544 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.543586 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37822: BlockingRecv error: recv got EOF from 127.0.0.1:37822 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:32:58.599498 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.581234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.2.147.253:37605)
0926 07:32:58.581405 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.581412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.592340 (+ 10928us) server_negotiation.cc:407] Connection header received
0926 07:32:58.593011 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.593018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.593114 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.593180 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.593926 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.595135 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.597169 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.597816 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.597916 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.598554 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.598698 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.598764 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.599012 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.599099 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:32:58.609575 25249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3df14706b4f4474e89305b9d55b4654b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3df14706b4f4474e89305b9d55b4654b" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 33643 } } peers { permanent_uuid: "599f0723f7ae479f9d9a1728f67cfede" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 37605 } } peers { permanent_uuid: "04823201883340ed9654d2292677fc2c" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 38211 } } }
I20260926 07:32:58.609906 25249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df14706b4f4474e89305b9d55b4654b [sys.catalog]: This master's current role is: LEADER
W20260926 07:32:58.637116 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.631941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.2.147.252:38211)
0926 07:32:58.632119 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.632127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.635478 (+ 3351us) server_negotiation.cc:407] Connection header received
0926 07:32:58.635687 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.635693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.635832 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.635956 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.636874 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37830: BlockingRecv error: recv got EOF from 127.0.0.1:37830 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:32:58.651060 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.644109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.2.147.253:37605)
0926 07:32:58.644339 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.644346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.644801 (+ 455us) server_negotiation.cc:407] Connection header received
0926 07:32:58.645085 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.645091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.645177 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.645259 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.645893 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.647069 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.649108 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.649783 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.649904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.650346 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.650475 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.650560 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.650713 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.650798 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260926 07:32:58.659651 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.657822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.2.147.252:38211)
0926 07:32:58.658062 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.658068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.658147 (+ 79us) server_negotiation.cc:407] Connection header received
0926 07:32:58.658391 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.658397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.658486 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.658591 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.659421 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 07:32:58.676633 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.671308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.2.147.253:37605)
0926 07:32:58.671460 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.671465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.671935 (+ 470us) server_negotiation.cc:407] Connection header received
0926 07:32:58.672138 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.672144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.672232 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.672334 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.672735 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.673524 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.674832 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.675337 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.675477 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.675869 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.675998 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.676102 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.676240 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.676329 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:32:58.683667 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.681994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.2.147.252:38211)
0926 07:32:58.682157 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.682163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.682447 (+ 284us) server_negotiation.cc:407] Connection header received
0926 07:32:58.682571 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.682576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.682667 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.682767 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.683475 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37852: BlockingRecv error: recv got EOF from 127.0.0.1:37852 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:32:58.708662 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.702826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.2.147.253:37605)
0926 07:32:58.703028 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.703034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.703348 (+ 314us) server_negotiation.cc:407] Connection header received
0926 07:32:58.703531 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.703537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.703634 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.703742 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.704103 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.705274 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.706591 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.707358 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.707491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.707860 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.708008 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.708101 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.708257 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.708391 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:32:58.720693 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.718734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.2.147.252:38211)
0926 07:32:58.718927 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.718933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.719290 (+ 357us) server_negotiation.cc:407] Connection header received
0926 07:32:58.719516 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.719521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.719637 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.719746 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.720446 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingRecv error: recv got EOF from 127.0.0.1:37864 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:58.761947 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.755863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.2.147.253:37605)
0926 07:32:58.756128 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.756135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.756531 (+ 396us) server_negotiation.cc:407] Connection header received
0926 07:32:58.756736 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.756740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.756826 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.756906 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.757396 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.758159 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.760148 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.760610 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.760781 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.761184 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.761290 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.761391 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.761548 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.761627 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 07:32:58.769531 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.767731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.2.147.252:38211)
0926 07:32:58.767926 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.767932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.768206 (+ 274us) server_negotiation.cc:407] Connection header received
0926 07:32:58.768400 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.768405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.768497 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.768599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.769302 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37868: BlockingRecv error: recv got EOF from 127.0.0.1:37868 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:32:58.842478 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.836568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.2.147.253:37605)
0926 07:32:58.836818 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.836828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.837378 (+ 550us) server_negotiation.cc:407] Connection header received
0926 07:32:58.837611 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.837623 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.837690 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.837789 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.838202 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.838949 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.840506 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.841184 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.841326 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.841710 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.841834 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.841939 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.842094 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.842172 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:32:58.853034 25232 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:58.848340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.2.147.252:38211)
0926 07:32:58.848557 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.848567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.851701 (+ 3134us) server_negotiation.cc:407] Connection header received
0926 07:32:58.851900 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.851906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.852000 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.852106 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:58.852858 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37876: BlockingRecv error: recv got EOF from 127.0.0.1:37876 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:32:58.911150 25246 catalog_manager.cc:1259] Loaded cluster ID: 183365f56710468da20be30d62abd928
I20260926 07:32:58.911329 25246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: loading cluster ID for follower catalog manager: success
I20260926 07:32:58.913926 25246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: acquiring CA information for follower catalog manager: success
I20260926 07:32:58.915591 25246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04823201883340ed9654d2292677fc2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:32:58.989785 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.985117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.2.147.253:37605)
0926 07:32:58.985294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.985299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.985656 (+ 357us) server_negotiation.cc:407] Connection header received
0926 07:32:58.985815 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.985820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.985893 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.985983 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.986324 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.987016 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.988266 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.988840 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.988937 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.989256 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:58.989359 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:58.989433 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:58.989522 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:32:58.989580 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:32:59.000803 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.995156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.2.147.252:38211)
0926 07:32:58.995393 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:58.995399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:58.995666 (+ 267us) server_negotiation.cc:407] Connection header received
0926 07:32:58.995865 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:58.995870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:58.996035 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:58.996158 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:58.996543 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.997256 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.998554 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:58.998966 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:58.999108 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:58.999675 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:59.000206 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:59.000375 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:59.000426 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:32:59.000534 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:32:59.014005 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.006952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.2.147.253:37605)
0926 07:32:59.007271 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:59.007279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:59.007879 (+ 600us) server_negotiation.cc:407] Connection header received
0926 07:32:59.008178 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:59.008187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:59.008283 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:59.008358 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:59.009208 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.010086 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.011916 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.012601 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.012696 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:59.013233 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:59.013345 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:59.013445 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:59.013603 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:32:59.013683 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 07:32:59.014997 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.006448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.2.147.252:38211)
0926 07:32:59.006614 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:59.006620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:59.008611 (+ 1991us) server_negotiation.cc:407] Connection header received
0926 07:32:59.008808 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:59.008814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:59.008889 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:59.008990 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:59.009643 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.010729 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.012906 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.013580 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.013692 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:59.014298 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:59.014402 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:59.014464 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:59.014678 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:32:59.014754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:32:59.015486 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.006269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.2.147.254:33643)
0926 07:32:59.008660 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:59.008665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:59.008683 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:59.008799 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:59.008806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:59.009036 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:59.009250 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:59.009882 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.010937 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.012722 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:59.013324 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:59.013951 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:59.014134 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:59.014626 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:59.015037 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:59.015100 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:32:59.015189 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2267,"thread_start_us":2165,"threads_started":1}
I20260926 07:32:59.020366 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25014
I20260926 07:32:59.021509 25076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:59.144825 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25014
W20260926 07:32:59.165494 25105 connection.cc:535] server connection from 127.0.0.1:56608 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:32:59.166034 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25088
I20260926 07:32:59.167050 25150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:59.274310 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25088
I20260926 07:32:59.294252 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25162
I20260926 07:32:59.295363 25224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:32:59.400633 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25162
2026-09-26T07:32:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4373 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:59 dist-test-slave-1dwd krb5kdc[25408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:59 dist-test-slave-1dwd krb5kdc[25408](info): set up 2 sockets
Sep 26 07:32:59 dist-test-slave-1dwd krb5kdc[25408](info): commencing operation
krb5kdc: starting...
W20260926 07:33:00.499944 2639 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/test-user.keytab.
Sep 26 07:33:00 dist-test-slave-1dwd krb5kdc[25408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
Entry for principal kudu/127.2.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab.
I20260926 07:33:00.758020 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38313
--webserver_interface=127.2.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:01.086599 25421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:33:01.086974 25421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:01.087098 25421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:01.087286 25421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:01.094424 25421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:01.094551 25421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:01.094645 25421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:01.094735 25421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:01.094817 25421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:01.106688 25421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.254.keytab
--principal=kudu/127.2.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.254:38313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:01.108593 25421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:01.109902 25421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:01.121569 25426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:01.121490 25429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:01.121536 25427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:01.122648 25421 server_base.cc:1034] running on GCE node
I20260926 07:33:01.123512 25421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:01.124920 25421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:01.126204 25421 hybrid_clock.cc:648] HybridClock initialized: now 1790407981126153 us; error 44 us; skew 500 ppm
I20260926 07:33:01.126536 25421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:01.129683 25421 init.cc:377] Logged in from keytab as kudu/127.2.147.254@KRBTEST.COM (short username kudu)
I20260926 07:33:01.131078 25421 webserver.cc:460] Webserver started at http://127.2.147.254:41363/ using document root <none> and password file <none>
I20260926 07:33:01.131827 25421 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:01.131971 25421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:01.132300 25421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:01.135339 25421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data/instance:
uuid: "158580ae605e4ad69dad818fd515fe74"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "0f60a7e66acb824fff5e9dc11de81641"
server_key_iv: "95eb959c1614100a21b298a5c940a3c5"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.136140 25421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance:
uuid: "158580ae605e4ad69dad818fd515fe74"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "0f60a7e66acb824fff5e9dc11de81641"
server_key_iv: "95eb959c1614100a21b298a5c940a3c5"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.141106 25421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 07:33:01.144484 25436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:01.145798 25421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:01.145993 25421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal
uuid: "158580ae605e4ad69dad818fd515fe74"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "0f60a7e66acb824fff5e9dc11de81641"
server_key_iv: "95eb959c1614100a21b298a5c940a3c5"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.146215 25421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:01.157708 25421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:01.161512 25421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:01.161782 25421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:01.180481 25421 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.254:38313
I20260926 07:33:01.180505 25487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.254:38313 every 8 connection(s)
I20260926 07:33:01.181897 25421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:01.184312 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25421
I20260926 07:33:01.184514 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-0/wal/instance
I20260926 07:33:01.184944 2639 external_mini_cluster.cc:1428] Setting key 254a8dcc40e1a865d574b7eb37c23c6b
I20260926 07:33:01.185827 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:01.192456 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.253@KRBTEST.COM" created.
I20260926 07:33:01.215322 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.196258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.2.147.254:38313)
0926 07:33:01.197880 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:01.197890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:01.197957 (+ 67us) server_negotiation.cc:407] Connection header received
0926 07:33:01.199508 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:01.199514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:01.199731 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:01.199967 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:01.201209 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.202417 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.203814 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.204475 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.204670 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.208086 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:01.208096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:01.208110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:01.208154 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:01.210974 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.211714 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.211722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.211727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.211867 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.212262 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.212267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.212270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.212735 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:01.212957 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:01.213462 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 07:33:01.213864 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1403,"thread_start_us":1203,"threads_started":1}
I20260926 07:33:01.215333 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.196258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38313 (local address 127.0.0.1:48292)
0926 07:33:01.197112 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.197154 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:01.199101 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:01.199975 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:01.199981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:01.200677 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:01.200996 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.201013 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.202567 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.202572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.203676 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.203687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.204698 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.204705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.204887 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.205574 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:01.205598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:01.207847 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:01.211103 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.211114 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.211125 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.211572 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.211985 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.211992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.211996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.212148 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.212866 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:01.212873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:01.213087 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:33:01.213333 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 07:33:01.213979 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":177,"thread_start_us":366,"threads_started":1}
I20260926 07:33:01.218170 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:33:01.219374 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.218910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57718)
0926 07:33:01.219075 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.219263 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:33:01.219645 25437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
I20260926 07:33:01.221180 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.220845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57726)
0926 07:33:01.220982 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.221054 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:33:01.221397 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:45389: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
Entry for principal kudu/127.2.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab.
I20260926 07:33:01.240177 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:45389
--webserver_interface=127.2.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:01.270754 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } attempt: 1
I20260926 07:33:01.272148 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.271587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57734)
0926 07:33:01.271819 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.271982 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:33:01.273195 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.272793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57736)
0926 07:33:01.272953 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.273035 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:33:01.273392 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:45389: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
I20260926 07:33:01.341681 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } attempt: 2
I20260926 07:33:01.342978 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.342373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57742)
0926 07:33:01.342673 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.342790 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:33:01.344116 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.343686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57746)
0926 07:33:01.343869 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.343954 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:33:01.344355 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:45389: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
I20260926 07:33:01.435603 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } attempt: 3
I20260926 07:33:01.436817 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.436312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57750)
0926 07:33:01.436539 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.436651 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:33:01.437915 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.437505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57756)
0926 07:33:01.437665 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.437749 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:33:01.438127 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:45389: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
W20260926 07:33:01.567077 25495 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:33:01.567492 25495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:01.567613 25495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:01.567749 25495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:01.574846 25495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:01.574929 25495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:01.574988 25495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:01.575062 25495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:01.575153 25495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:01.581380 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } attempt: 4
I20260926 07:33:01.582604 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.582075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57770)
0926 07:33:01.582386 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.582474 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 07:33:01.583667 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.583312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57780)
0926 07:33:01.583458 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.583551 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:33:01.583906 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.253:45389: Network error: Client connection negotiation failed: client connection to 127.2.147.253:45389: connect: Connection refused (error 111)
I20260926 07:33:01.586476 25495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.253.keytab
--principal=kudu/127.2.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.253:45389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:01.587880 25495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:01.589174 25495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:01.600746 25500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:01.600771 25501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:01.600863 25503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:01.601714 25495 server_base.cc:1034] running on GCE node
I20260926 07:33:01.602505 25495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:01.604018 25495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:01.605260 25495 hybrid_clock.cc:648] HybridClock initialized: now 1790407981605192 us; error 42 us; skew 500 ppm
I20260926 07:33:01.605587 25495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:01.608757 25495 init.cc:377] Logged in from keytab as kudu/127.2.147.253@KRBTEST.COM (short username kudu)
I20260926 07:33:01.610167 25495 webserver.cc:460] Webserver started at http://127.2.147.253:35811/ using document root <none> and password file <none>
I20260926 07:33:01.610801 25495 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:01.610888 25495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:01.611147 25495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:01.614027 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data/instance:
uuid: "7e21003d14ae422997a8d3c8ceca5bf2"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "343e3662f1471c947139eee950b32cc8"
server_key_iv: "28450cefd7c560a96adf97265faf3212"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.614720 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal/instance:
uuid: "7e21003d14ae422997a8d3c8ceca5bf2"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "343e3662f1471c947139eee950b32cc8"
server_key_iv: "28450cefd7c560a96adf97265faf3212"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.619522 25495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:33:01.622695 25510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:01.624004 25495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:01.624138 25495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal
uuid: "7e21003d14ae422997a8d3c8ceca5bf2"
format_stamp: "Formatted at 2026-09-26 07:33:01 on dist-test-slave-1dwd"
server_key: "343e3662f1471c947139eee950b32cc8"
server_key_iv: "28450cefd7c560a96adf97265faf3212"
server_key_version: "encryptionkey@0"
I20260926 07:33:01.624298 25495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:01.653848 25495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:01.658538 25495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:01.658772 25495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:01.677310 25495 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.253:45389
I20260926 07:33:01.677317 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.253:45389 every 8 connection(s)
I20260926 07:33:01.678745 25495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/data/info.pb
I20260926 07:33:01.682441 25562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:01.687363 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25495
I20260926 07:33:01.687539 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-1/wal/instance
I20260926 07:33:01.687981 2639 external_mini_cluster.cc:1428] Setting key 1e141c48db6d36be5b13c4c37a9906e2
I20260926 07:33:01.689255 25562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:33:01.710768 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.692007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.2.147.254:38313)
0926 07:33:01.692200 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:01.692208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:01.693881 (+ 1673us) server_negotiation.cc:407] Connection header received
0926 07:33:01.695890 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:01.695896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:01.695992 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:01.696105 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:01.697622 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.698299 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.699891 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.700289 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.700436 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.704027 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:01.704036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:01.704043 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:01.704088 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:01.707754 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.708477 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.708486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.708492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.708639 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.709058 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.709062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.709064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.709339 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:01.709512 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:01.710210 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 07:33:01.710436 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:33:01.711812 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.692645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38313 (local address 127.0.0.1:48296)
0926 07:33:01.693718 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.693755 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:01.695611 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:01.696349 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:01.696357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:01.696873 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:01.697442 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.697459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.698457 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.698462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.699738 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.699750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.700519 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.700527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.700753 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.701450 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:01.701474 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:01.703793 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:01.707889 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.707896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.707909 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.708342 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.708747 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.708750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.708752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.708919 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.709483 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:01.709490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:01.709828 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 07:33:01.710104 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 07:33:01.710635 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":617,"threads_started":1}
WARNING: no policy specified for kudu/127.2.147.252@KRBTEST.COM; defaulting to no policy
I20260926 07:33:01.713603 25562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.147.252@KRBTEST.COM" created.
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:33:01.728109 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.714328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57784)
0926 07:33:01.714518 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.714542 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:01.714648 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:01.715716 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:01.715720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:01.716159 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:01.716288 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.716295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.717812 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.717818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.718658 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.718669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.719555 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.719560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.719746 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.720347 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:01.720363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:01.722525 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:01.725647 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.725654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.725661 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.726009 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.726388 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.726392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.726396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.726542 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.727105 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:01.727110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:01.727266 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 07:33:01.727417 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:33:01.727713 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:33:01.728317 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.714567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.2.147.253:45389)
0926 07:33:01.715271 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:01.715284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:01.715326 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:33:01.715379 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:01.715386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:01.715579 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:01.715728 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:01.716501 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.717677 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.718790 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.719408 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.719535 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.722717 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:01.722727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:01.722739 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:01.722769 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:01.725520 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.726151 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.726161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.726167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.726277 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.726643 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.726650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.726653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.726974 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:01.727180 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:01.727577 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 07:33:01.727826 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":473,"thread_start_us":343,"threads_started":1}
I20260926 07:33:01.730756 25562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:33:01.732113 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.731539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49670)
0926 07:33:01.731753 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.731918 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 07:33:01.732376 25513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:01.734037 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.733607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49678)
0926 07:33:01.733782 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.733895 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 07:33:01.734261 25562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
Entry for principal kudu/127.2.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab.
I20260926 07:33:01.746151 2639 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
/tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:36837
--webserver_interface=127.2.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.147.212:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:01.783576 25562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 1
I20260926 07:33:01.784862 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.784378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49680)
0926 07:33:01.784565 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.784671 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 07:33:01.785864 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.785498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49686)
0926 07:33:01.785666 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.785761 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:33:01.786067 25562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:01.854336 25562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 2
I20260926 07:33:01.855482 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.854997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49698)
0926 07:33:01.855185 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.855327 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:33:01.856509 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.856215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49706)
0926 07:33:01.856343 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.856407 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:33:01.856813 25562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:01.883276 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } attempt: 5
Sep 26 07:33:01 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:33:01.903751 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.884146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.2.147.253:45389)
0926 07:33:01.884336 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:01.884358 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:01.884380 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:33:01.884576 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:01.884585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:01.884650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:01.884754 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:01.885520 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.886205 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.887399 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.887839 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.887981 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.898231 (+ 10250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:01.898237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:01.898240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:01.898274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:01.900881 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.901809 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.901813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.901815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.901926 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:01.902373 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:01.902376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:01.902377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:01.902643 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:01.902748 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:01.903337 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 07:33:01.903508 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:33:01.904099 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.884053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57792)
0926 07:33:01.884269 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.884290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:01.884436 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:01.884791 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:01.884795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:01.885193 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:01.885363 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.885371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.886329 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.886337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.887189 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:01.887251 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:01.887991 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:01.887996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:01.888227 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:01.895600 (+ 7373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:01.895621 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:01.898005 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:01.901036 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.901043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.901050 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.901690 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.902042 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:01.902048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:01.902054 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:01.902259 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:01.902780 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:01.902787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:01.903001 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:33:01.903264 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 07:33:01.903622 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:33:01.905210 25488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:33:01.906301 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.905936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49716)
0926 07:33:01.906098 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.906172 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:33:01.907188 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.906872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49722)
0926 07:33:01.907017 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.907096 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:33:01.907442 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:01.948028 25562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 3
I20260926 07:33:01.949205 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.948742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49734)
0926 07:33:01.948957 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.949049 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 07:33:01.950246 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.949809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49744)
0926 07:33:01.950024 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.950143 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 07:33:01.950469 25562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:01.958632 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 1
I20260926 07:33:01.959713 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.959299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49754)
0926 07:33:01.959475 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.959560 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 07:33:01.960686 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.960365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49770)
0926 07:33:01.960521 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.960586 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:33:01.960884 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:02.030352 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 2
I20260926 07:33:02.031824 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.031145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49784)
0926 07:33:02.031456 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.031578 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 07:33:02.032934 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.032520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49786)
0926 07:33:02.032702 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.032785 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 07:33:02.033331 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
W20260926 07:33:02.079866 25569 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:33:02.080186 25569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:02.080317 25569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:02.080440 25569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:02.087827 25569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:02.087916 25569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:02.088004 25569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:02.088073 25569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:02.088196 25569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:02.093684 25562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 4
I20260926 07:33:02.094982 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.094432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49798)
0926 07:33:02.094690 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.094791 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:33:02.096082 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.095714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49812)
0926 07:33:02.095866 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.095946 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:33:02.096391 25562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:02.099288 25569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.147.212:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.147.254:38313,127.2.147.253:45389,127.2.147.252:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/krb5kdc/kudu_127.2.147.252.keytab
--principal=kudu/127.2.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.147.252:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 06:26:09 UTC on ccycloud.quasar-arzrrx.root.comops.site
build id 4673740
ASAN enabled
I20260926 07:33:02.100621 25569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:02.101940 25569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:02.115015 25575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:02.115039 25574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:02.115671 25577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:02.116014 25569 server_base.cc:1034] running on GCE node
I20260926 07:33:02.116828 25569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:02.118425 25569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:02.119652 25569 hybrid_clock.cc:648] HybridClock initialized: now 1790407982119589 us; error 38 us; skew 500 ppm
I20260926 07:33:02.119982 25569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:02.123764 25569 init.cc:377] Logged in from keytab as kudu/127.2.147.252@KRBTEST.COM (short username kudu)
I20260926 07:33:02.125671 25569 webserver.cc:460] Webserver started at http://127.2.147.252:39611/ using document root <none> and password file <none>
I20260926 07:33:02.126650 25569 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:02.126773 25569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:02.127146 25569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:02.131907 25569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data/instance:
uuid: "eeae7e58d9614686a82c7140209c3e20"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-1dwd"
server_key: "c22de697857eb85cdc0c6dcfacae225c"
server_key_iv: "c92f6fbac8c1ea5ddba24ccdee3d21d7"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.133025 25569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal/instance:
uuid: "eeae7e58d9614686a82c7140209c3e20"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-1dwd"
server_key: "c22de697857eb85cdc0c6dcfacae225c"
server_key_iv: "c92f6fbac8c1ea5ddba24ccdee3d21d7"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.139598 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 3
I20260926 07:33:02.140702 25569 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260926 07:33:02.140995 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.140267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49814)
0926 07:33:02.140516 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.140634 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 07:33:02.142309 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.141792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49822)
0926 07:33:02.142035 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.142131 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 07:33:02.142608 25488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.147.252:36837: Network error: Client connection negotiation failed: client connection to 127.2.147.252:36837: connect: Connection refused (error 111)
I20260926 07:33:02.144953 25584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:02.146441 25569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:02.146626 25569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal
uuid: "eeae7e58d9614686a82c7140209c3e20"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-1dwd"
server_key: "c22de697857eb85cdc0c6dcfacae225c"
server_key_iv: "c92f6fbac8c1ea5ddba24ccdee3d21d7"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.146852 25569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:02.161276 25569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:02.165512 25569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:02.165781 25569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:02.185086 25569 rpc_server.cc:307] RPC server started. Bound to: 127.2.147.252:36837
I20260926 07:33:02.185173 25635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.147.252:36837 every 8 connection(s)
I20260926 07:33:02.186461 25569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/data/info.pb
I20260926 07:33:02.190512 2639 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu as pid 25569
I20260926 07:33:02.190697 2639 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNb1Qs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407623164223-2639-0/minicluster-data/master-2/wal/instance
I20260926 07:33:02.191138 2639 external_mini_cluster.cc:1428] Setting key e807ccbdaf549276f62647e586840876
I20260926 07:33:02.191589 25636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:02.201673 25636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
I20260926 07:33:02.217803 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.192990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.2.147.252:36837)
0926 07:33:02.193825 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.193840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.193887 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:33:02.195737 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.195766 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.196297 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.196739 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.198664 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.200661 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.202914 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.203871 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.204066 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.209361 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.209374 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.209390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.209458 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.213376 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.214128 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.214136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.214141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.214304 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.214694 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.214703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.214705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.215394 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.215711 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.216401 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.216826 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":361,"threads_started":1}
I20260926 07:33:02.224604 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.204526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.2.147.254:38313)
0926 07:33:02.204811 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.204819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.205385 (+ 566us) server_negotiation.cc:407] Connection header received
0926 07:33:02.205668 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.205677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.205774 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.205890 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.207344 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.208083 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.209454 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.210365 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.210516 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.215102 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.215109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.215114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.215154 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.220946 (+ 5792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.221890 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.221898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.221904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.222109 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.222682 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.222690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.222694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.223121 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.223308 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.223943 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.224178 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 07:33:02.224746 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.204416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.254:38313 (local address 127.0.0.1:48310)
0926 07:33:02.205253 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.205296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:02.205511 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:02.206001 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:02.206011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:02.206818 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:02.207179 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.207221 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.208245 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.208250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.209291 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.209304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.210570 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.210580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.210792 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.211636 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:02.211663 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:02.214882 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:02.221077 (+ 6195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.221090 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.221098 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.221707 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.222290 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.222298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.222303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.222504 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.223295 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:02.223306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:02.223545 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 07:33:02.223813 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 07:33:02.224190 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"spinlock_wait_cycles":21888,"thread_start_us":418,"threads_started":1}
I20260926 07:33:02.226397 25636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.254@KRBTEST.COM
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:33:02.241964 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.221459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.2.147.254:38313)
0926 07:33:02.222110 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.222114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.222133 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:33:02.222174 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.222177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.222228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.222344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.223785 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.225204 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.226736 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.227614 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.227756 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.233189 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.233198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.233206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.233252 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.237655 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.238533 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.238541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.238547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.238714 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.239132 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.239139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.239144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.239590 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.239909 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.241142 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.241416 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":300,"threads_started":1}
I20260926 07:33:02.242108 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.227452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.2.147.253:45389)
0926 07:33:02.227631 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.227636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.228188 (+ 552us) server_negotiation.cc:407] Connection header received
0926 07:33:02.228732 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.228738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.228831 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.228975 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.230025 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.231116 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.232423 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.232823 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.232971 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.236409 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.236414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.236417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.236449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.239520 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.240141 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.240151 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.240153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.240277 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.240659 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.240663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.240665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.240939 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.241089 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.241541 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.241738 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:33:02.242810 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.227244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.253:45389 (local address 127.0.0.1:57794)
0926 07:33:02.228065 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.228088 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:02.228596 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:02.229055 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:02.229063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:02.229607 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:02.229851 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.229863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.231285 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.231290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.232277 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.232288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.232985 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.232991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.233178 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.233998 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:02.234015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:02.236219 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:02.239643 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.239649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.239654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.240016 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.240390 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.240394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.240398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.240537 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.241155 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:02.241158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:02.241283 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 07:33:02.241430 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:33:02.241700 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677}
I20260926 07:33:02.244063 25636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.252@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.147.253@KRBTEST.COM
I20260926 07:33:02.260047 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.244903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49826)
0926 07:33:02.245105 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.245119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:02.245241 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:02.245527 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:02.245531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:02.245902 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:02.246028 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.246035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.247058 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.247066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.248776 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.248794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.249948 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.249954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.250188 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.250883 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:02.250943 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:02.253111 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:02.257206 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.257210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.257213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.257553 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.257944 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.257950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.257953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.258109 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.258867 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:02.258874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:02.259088 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:33:02.259317 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 07:33:02.259618 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 07:33:02.260138 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.244909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.2.147.252:36837)
0926 07:33:02.245057 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.245062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.245193 (+ 131us) server_negotiation.cc:407] Connection header received
0926 07:33:02.245358 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.245363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.245423 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.245513 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.246153 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.246913 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.248930 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.249576 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.249695 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.253276 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.253282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.253286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.253326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.257067 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.257694 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.257701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.257704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.257833 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.258243 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.258251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.258257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.258689 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.258907 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.259398 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.259794 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":51}
I20260926 07:33:02.264936 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.244249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.2.147.253:45389)
0926 07:33:02.244414 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.244419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.244495 (+ 76us) server_negotiation.cc:407] Connection header received
0926 07:33:02.244688 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.244696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.244786 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.244898 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.246806 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.247649 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.248859 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.249538 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.249728 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.257301 (+ 7573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.257310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.257317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.257367 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.261651 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.262527 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.262536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.262542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.262738 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.263164 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.263172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.263177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.263600 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.263936 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.264200 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.264478 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:33:02.265895 25636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: Bootstrap starting.
I20260926 07:33:02.269670 25636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:02.270499 2639 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:33:02.270814 25636 log.cc:824] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:02.273447 25636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: No bootstrap required, opened a new log
I20260926 07:33:02.274123 25463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "158580ae605e4ad69dad818fd515fe74"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48314
I20260926 07:33:02.280475 25636 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.280812 25636 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeae7e58d9614686a82c7140209c3e20, State: Initialized, Role: FOLLOWER
I20260926 07:33:02.281239 25636 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.282168 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.282554 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.283314 25636 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:33:02.293121 25654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:33:02.293242 25654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:33:02.291056 25569 master_runner.cc:186] Error getting master registration for 127.2.147.254:38313: OK, instance_id { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" instance_seqno: 1790407981174846 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:33:02.302533 25569 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:33:02.319796 25488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 4
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.254@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:33:02.335119 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.320623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.2.147.252:36837)
0926 07:33:02.320825 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.320831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.320864 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:33:02.320997 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.321002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.321118 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.321236 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.321998 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.322743 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.324074 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.324541 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.324670 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.328400 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.328406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.328411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.328451 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.332213 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.333031 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.333036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.333039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.333169 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.333580 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.333587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.333591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.333948 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.334112 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.334534 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.334752 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:33:02.335125 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.320523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49832)
0926 07:33:02.320723 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.320746 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:02.320874 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:02.321280 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:02.321287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:02.321687 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:02.321839 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.321849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.322901 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.322907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.323935 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.323946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.324697 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.324703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.324901 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.325558 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:02.325576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:02.328185 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:02.332357 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.332363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.332368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.332909 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.333285 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.333290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.333293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.333468 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.334082 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:02.334087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:02.334265 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 07:33:02.334440 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 07:33:02.334724 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:33:02.341394 25488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: Bootstrap starting.
I20260926 07:33:02.344660 25488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:02.345997 25488 log.cc:824] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:02.348733 25488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: No bootstrap required, opened a new log
I20260926 07:33:02.354266 25488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.354602 25488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158580ae605e4ad69dad818fd515fe74, State: Initialized, Role: FOLLOWER
I20260926 07:33:02.355046 25488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.355984 25657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.356393 25657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.357127 25488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:02.357267 25463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:33:02.357491 25463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:02.360989 25463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.363430 25611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "158580ae605e4ad69dad818fd515fe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eeae7e58d9614686a82c7140209c3e20"
I20260926 07:33:02.363909 25611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:02.364742 25463 leader_election.cc:288] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [CANDIDATE]: Term 1 election: Requested vote from peers 7e21003d14ae422997a8d3c8ceca5bf2 (127.2.147.253:45389), eeae7e58d9614686a82c7140209c3e20 (127.2.147.252:36837)
I20260926 07:33:02.367283 25611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158580ae605e4ad69dad818fd515fe74 in term 1.
I20260926 07:33:02.367995 25439 leader_election.cc:302] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158580ae605e4ad69dad818fd515fe74, eeae7e58d9614686a82c7140209c3e20; no voters:
I20260926 07:33:02.368551 25657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:02.369253 25657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [term 1 LEADER]: Becoming Leader. State: Replica: 158580ae605e4ad69dad818fd515fe74, State: Running, Role: LEADER
I20260926 07:33:02.369760 25657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.379433 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158580ae605e4ad69dad818fd515fe74. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.379869 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:02.395036 25672 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:33:02.395588 25562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } attempt: 5
I20260926 07:33:02.395910 25678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:02.401021 25421 master_runner.cc:186] Error getting master registration for 127.2.147.253:45389: OK, instance_id { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" instance_seqno: 1790407981671982 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:33:02.414105 25421 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:33:02.414516 25442 tablet.cc:2369] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:33:02.417941 25590 tablet.cc:2369] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:02.419904 25678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:02.432761 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.385303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.2.147.253:45389)
0926 07:33:02.385491 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.385497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.398342 (+ 12845us) server_negotiation.cc:407] Connection header received
0926 07:33:02.399191 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.399197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.399296 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.399364 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.401502 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.402629 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.404872 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.405558 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.405657 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.425381 (+ 19724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.425389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.425395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.425443 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.429125 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.429973 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.429980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.429985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.430179 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.430619 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.430624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.430628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.430988 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.431281 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.431960 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.432255 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:33:02 dist-test-slave-1dwd krb5kdc[25408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.147.253@KRBTEST.COM for kudu/127.2.147.252@KRBTEST.COM
I20260926 07:33:02.448690 25611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [term 1 FOLLOWER]: Refusing update from remote peer 158580ae605e4ad69dad818fd515fe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:02.449793 25670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:02.464872 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.385254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.2.147.254:38313)
0926 07:33:02.385548 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.385555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.398912 (+ 13357us) server_negotiation.cc:407] Connection header received
0926 07:33:02.399021 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.399026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.399099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.399194 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.411421 (+ 12227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.412530 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.422186 (+ 9656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.422989 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.423110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.446503 (+ 23393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.446509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.446512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.446550 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.453052 (+ 6502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.454839 (+ 1787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.454844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.454848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.454991 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.457041 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.457050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.457055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.457529 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.457827 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.464060 (+ 6233us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.464390 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 07:33:02.473240 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.385686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.2.147.252:36837)
0926 07:33:02.385900 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.385906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.393400 (+ 7494us) server_negotiation.cc:407] Connection header received
0926 07:33:02.393637 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.393646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.393757 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.393887 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.415500 (+ 21613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.416650 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.434057 (+ 17407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.434698 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.434796 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.458466 (+ 23670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.458475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.458482 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.458534 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.462893 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.470176 (+ 7283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.470185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.470191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.470368 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.470902 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.470910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.470916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.471405 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.471715 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.472410 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.472701 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:33:02.479135 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.396224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.147.252:36837 (local address 127.0.0.1:49844)
0926 07:33:02.396479 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:02.396498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:02.396607 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:02.418990 (+ 22383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:02.418996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:02.419538 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:02.419756 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.419771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.435870 (+ 16099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.435878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.437679 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:02.437698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.442290 (+ 4592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.442296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:02.442535 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.443106 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:02.443129 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:02.445857 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:02.475464 (+ 29607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.475475 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.475481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.476174 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.476762 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:02.476770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:02.476775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:02.476993 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:02.477846 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:02.477854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:02.478049 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 07:33:02.478224 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 07:33:02.478609 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 07:33:02.480883 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.397201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.2.147.252:36837)
0926 07:33:02.415050 (+ 17849us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.415056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.415080 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:33:02.415139 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.415145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.415246 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.419916 (+ 4670us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.420755 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.421993 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.437912 (+ 15919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.438811 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.446023 (+ 7212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.446290 (+ 267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.446298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.446304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.446348 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.469975 (+ 23627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.476376 (+ 6401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.476384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.476390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.476611 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.477169 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.477176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.477180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.477681 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.479895 (+ 2214us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.480047 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.480331 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17681,"thread_start_us":18055,"threads_started":1}
I20260926 07:33:02.489832 25562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: Bootstrap starting.
I20260926 07:33:02.494006 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158580ae605e4ad69dad818fd515fe74. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.494379 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.496471 25562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:02.498075 25681 mvcc.cc:205] Tried to move back new op lower bound from 7333511096092602368 to 7333511095790878720. Current Snapshot: MvccSnapshot[applied={T|T < 7333511096092602368}]
I20260926 07:33:02.500715 25562 log.cc:824] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:02.525475 25678 catalog_manager.cc:1347] Generated new cluster ID: 9fcfc7849f554302bb04b21653808ced
I20260926 07:33:02.525610 25678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:02.527486 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.527882 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:02.531723 25562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: No bootstrap required, opened a new log
I20260926 07:33:02.540828 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.541090 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.543498 25562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.543830 25562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e21003d14ae422997a8d3c8ceca5bf2, State: Initialized, Role: FOLLOWER
I20260926 07:33:02.544332 25562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } }
I20260926 07:33:02.549117 25687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.549571 25687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.552726 25562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:02.553192 25536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:02.557103 25536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [term 1 FOLLOWER]: Refusing update from remote peer 158580ae605e4ad69dad818fd515fe74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:02.557746 25537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "158580ae605e4ad69dad818fd515fe74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e21003d14ae422997a8d3c8ceca5bf2"
I20260926 07:33:02.559741 25657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:02.561803 25537 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158580ae605e4ad69dad818fd515fe74 in term 1.
I20260926 07:33:02.595739 25700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.596148 25700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.596560 25687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158580ae605e4ad69dad818fd515fe74. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.596849 25687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:33:02.621802 25702 catalog_manager.cc:1259] Loaded cluster ID: 9fcfc7849f554302bb04b21653808ced
I20260926 07:33:02.621935 25702 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: loading cluster ID for follower catalog manager: success
W20260926 07:33:02.623014 25702 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 07:33:02.623139 25702 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:33:02.644775 25657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "158580ae605e4ad69dad818fd515fe74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158580ae605e4ad69dad818fd515fe74" member_type: VOTER last_known_addr { host: "127.2.147.254" port: 38313 } } peers { permanent_uuid: "7e21003d14ae422997a8d3c8ceca5bf2" member_type: VOTER last_known_addr { host: "127.2.147.253" port: 45389 } } peers { permanent_uuid: "eeae7e58d9614686a82c7140209c3e20" member_type: VOTER last_known_addr { host: "127.2.147.252" port: 36837 } } }
I20260926 07:33:02.645116 25657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74 [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:02.661746 25678 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:02.663995 25678 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:02.700228 25678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158580ae605e4ad69dad818fd515fe74: Generated new TSK 0
I20260926 07:33:02.701439 25678 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:33:02.840093 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.815534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.2.147.253:45389)
0926 07:33:02.815763 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.815770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.836705 (+ 20935us) server_negotiation.cc:407] Connection header received
0926 07:33:02.838425 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.838432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.838524 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.838603 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.839816 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57818: BlockingRecv error: recv got EOF from 127.0.0.1:57818 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:33:02.845329 25641 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.833967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.2.147.254:38313)
0926 07:33:02.834166 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.834175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.843512 (+ 9337us) server_negotiation.cc:407] Connection header received
0926 07:33:02.843924 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.843930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.844038 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.844115 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.845082 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48328: BlockingRecv error: recv got EOF from 127.0.0.1:48328 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 07:33:02.849305 25679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.822067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.2.147.252:36837)
0926 07:33:02.822244 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.822250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.843010 (+ 20760us) server_negotiation.cc:407] Connection header received
0926 07:33:02.847770 (+ 4760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.847777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.847896 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.847973 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.849069 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49854: BlockingRecv error: recv got EOF from 127.0.0.1:49854 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:33:02.895758 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.882699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.2.147.253:45389)
0926 07:33:02.882902 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.882909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.893508 (+ 10599us) server_negotiation.cc:407] Connection header received
0926 07:33:02.894094 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.894102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.894230 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.894312 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.895476 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57828: BlockingRecv error: recv got EOF from 127.0.0.1:57828 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 07:33:02.945227 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.932190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.2.147.253:45389)
0926 07:33:02.932392 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.932399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.943146 (+ 10747us) server_negotiation.cc:407] Connection header received
0926 07:33:02.943688 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.943695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.943813 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.943895 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.944989 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57834: BlockingRecv error: recv got EOF from 127.0.0.1:57834 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:33:02.959854 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.957474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.2.147.253:45389)
0926 07:33:02.957676 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.957683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.958298 (+ 615us) server_negotiation.cc:407] Connection header received
0926 07:33:02.958499 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.958504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.958626 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.958731 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.959609 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57836: BlockingRecv error: recv got EOF from 127.0.0.1:57836 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:33:02.969991 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.967966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.2.147.253:45389)
0926 07:33:02.968222 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.968228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.968329 (+ 101us) server_negotiation.cc:407] Connection header received
0926 07:33:02.968627 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.968633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.968737 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.968853 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.969737 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57844: BlockingRecv error: recv got EOF from 127.0.0.1:57844 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260926 07:33:02.984160 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:02.982146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.2.147.253:45389)
0926 07:33:02.982322 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.982329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.982705 (+ 376us) server_negotiation.cc:407] Connection header received
0926 07:33:02.982893 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.982899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.983001 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.983115 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.983923 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57854: BlockingRecv error: recv got EOF from 127.0.0.1:57854 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 07:33:03.007004 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:03.004681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.2.147.253:45389)
0926 07:33:03.004869 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.004876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.005460 (+ 584us) server_negotiation.cc:407] Connection header received
0926 07:33:03.005665 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.005671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.005772 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.005885 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.006759 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57864: BlockingRecv error: recv got EOF from 127.0.0.1:57864 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:33:03.054073 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:03.052090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.2.147.253:45389)
0926 07:33:03.052315 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.052338 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.052665 (+ 327us) server_negotiation.cc:407] Connection header received
0926 07:33:03.052838 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.052849 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.052994 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.053125 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.053794 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57876: BlockingRecv error: recv got EOF from 127.0.0.1:57876 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 07:33:03.122749 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:03.121055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.2.147.253:45389)
0926 07:33:03.121235 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.121240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.121621 (+ 381us) server_negotiation.cc:407] Connection header received
0926 07:33:03.121799 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.121803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.121887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.121982 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.122587 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57886: BlockingRecv error: recv got EOF from 127.0.0.1:57886 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 07:33:03.256045 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:03.254142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.2.147.253:45389)
0926 07:33:03.254319 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.254324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.254734 (+ 410us) server_negotiation.cc:407] Connection header received
0926 07:33:03.254892 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.254896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.254966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.255057 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.255799 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57888: BlockingRecv error: recv got EOF from 127.0.0.1:57888 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:33:03.295715 25654 catalog_manager.cc:1259] Loaded cluster ID: 9fcfc7849f554302bb04b21653808ced
I20260926 07:33:03.295840 25654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: loading cluster ID for follower catalog manager: success
I20260926 07:33:03.298236 25654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: acquiring CA information for follower catalog manager: success
I20260926 07:33:03.300004 25654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eeae7e58d9614686a82c7140209c3e20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:33:03.522722 25566 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:03.521046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.2.147.253:45389)
0926 07:33:03.521272 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.521283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.521438 (+ 155us) server_negotiation.cc:407] Connection header received
0926 07:33:03.521598 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.521601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.521688 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.521776 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.522511 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57896: BlockingRecv error: recv got EOF from 127.0.0.1:57896 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:33:03.626466 25702 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: acquiring CA information for follower catalog manager: success
I20260926 07:33:03.628420 25702 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e21003d14ae422997a8d3c8ceca5bf2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:33:04.044734 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:04.039337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.2.147.253:45389)
0926 07:33:04.039988 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:04.039992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:04.040012 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:33:04.040148 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:04.040152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:04.040302 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:04.040409 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:04.040745 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.041467 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.042723 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.043129 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.043254 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:04.043713 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:04.044227 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:04.044337 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:04.044447 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:33:04.044510 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":451,"threads_started":1}
I20260926 07:33:04.056888 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:04.050738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.2.147.252:36837)
0926 07:33:04.051276 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:04.051282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:04.051377 (+ 95us) server_negotiation.cc:407] Connection header received
0926 07:33:04.051630 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:04.051634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:04.051845 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:04.052021 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:04.052424 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.053143 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.054512 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.054917 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.055087 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:04.055636 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:04.056228 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:04.056418 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:04.056468 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 07:33:04.056548 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
I20260926 07:33:04.070516 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:04.063743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.2.147.252:36837)
0926 07:33:04.064346 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:04.064354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:04.064380 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:33:04.064667 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:04.064676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:04.064788 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:04.064865 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:04.065591 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.066767 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.068605 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.069033 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.069168 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:04.069765 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:04.069892 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:04.069994 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:04.070126 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:33:04.070207 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260926 07:33:04.071120 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:04.063574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.2.147.253:45389)
0926 07:33:04.063793 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:04.063803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:04.064949 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 07:33:04.065141 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:04.065149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:04.065253 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:04.065342 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:04.066066 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.067141 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.069129 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.069727 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.069858 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:04.070339 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:04.070478 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:04.070544 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:04.070783 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:33:04.070877 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:33:04.072309 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:04.062863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.2.147.254:38313)
0926 07:33:04.065454 (+ 2591us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:04.065461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:04.065478 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:33:04.065591 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:04.065596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:04.065883 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:04.066082 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:04.066661 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.067760 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.069646 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:04.070252 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:04.070367 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:04.071014 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:04.071491 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:04.071593 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:04.072046 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 07:33:04.072110 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2467,"thread_start_us":2349,"threads_started":1}
I20260926 07:33:04.076176 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25421
I20260926 07:33:04.077356 25483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:04.192039 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25421
I20260926 07:33:04.212961 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25495
I20260926 07:33:04.214084 25557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:04.322575 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25495
I20260926 07:33:04.342832 2639 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu pid 25569
I20260926 07:33:04.344031 25631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 07:33:04.450696 2639 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNb1Qs/build/asan/bin/kudu with pid 25569
2026-09-26T07:33:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5048 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9421 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361249 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST